Why LLM Observability Matters: How We Measure Every Query Inside MMMGPT

If you cannot measure your AI, you cannot improve it.

At Aryma Labs, we have always believed that good decisions begin with good measurement. It is a principle that underpins Marketing Mix Modeling, experimentation and every recommendation we make to our clients. Decisions backed by evidence consistently outperform decisions backed by intuition.

As we built MMMGPT, our conversational AI for Marketing Mix Modeling, Experimentation and Causality; we realised the same principle applies to AI itself. An LLM application is only as good as your ability to understand what happens between a user's question and the model's response. Without that visibility, every improvement becomes a guess.

Why traditional monitoring falls short

Traditional software applications usually fail loudly. A service goes down, an API returns an error, or infrastructure monitoring immediately tells you where the problem occurred. When something breaks, engineers know where to look.

LLM applications behave differently.

The application responds successfully. The answer is well written. The latency looks normal. Yet the response may still be incorrect because the model retrieved the wrong context or reasoned incorrectly over the right one. Nothing crashes, no alert is triggered, and from a monitoring perspective, everything appears healthy.

The only indication that something has gone wrong is often the user quietly losing confidence in the system. They stop using the product long before an issue appears on an engineering dashboard.

This is why production AI systems need something beyond traditional application monitoring. They need observability.

What LLM observability actually means

LLM observability is the ability to inspect every stage of an interaction with a language model. Instead of treating the model as a black box, every request becomes traceable from beginning to end.

For a Retrieval Augmented Generation (RAG) application, observability means being able to inspect every stage of that journey.

Figure 1. What Langfuse records for a single interaction, end to end, including the exact context retrieved from the knowledge base before the model generated its answer.

When every interaction is recorded, debugging shifts from assumptions to evidence. Rather than wondering why an answer was generated, teams can inspect exactly what happened and make improvements based on real data.

Bringing observability into MMMGPT

MMMGPT is designed to help users explore Marketing Mix Modeling, causality and experimentation through natural language conversations. As we continued to improve the product, we wanted greater visibility into every interaction without adding unnecessary complexity to our engineering stack.

To achieve that, we integrated Langfuse into MMMGPT.

Figure 2. MMMGPT provides a conversational interface for Marketing Mix Modeling, experimentation and causality. Every interaction is traced using Langfuse.

Langfuse is an open source LLM observability platform that records each interaction end to end, the user's question, the context retrieved for it, and the final generated response. Each traced interaction records:

• User question
• Retrieved context
• Generated response
• Input and output tokens
• Latency
• Cost per query
• User and session metadata

Figure 3. Langfuse groups every recorded interaction by workflow. Across MMMGPT this covers model generations, translation events and custom application events, giving a single view of where activity is concentrated.

Each interaction receives a unique trace. If a user reports that MMMGPT produced an unexpected answer, we no longer need to recreate the scenario manually. Instead, we can inspect the exact interaction, review the retrieved context and understand what the model saw before generating its response.

Separating retrieval problems from reasoning problems

One design decision has had a significant impact on how we debug MMMGPT.

We log the retrieved context alongside every trace.

This allows us to distinguish between two fundamentally different failure modes.

The first is a retrieval issue. The model never received the information it needed because the retrieval layer surfaced the wrong documents.

The second is a reasoning issue. The correct context was retrieved, but the model failed to reason over it effectively.

These problems require completely different solutions. Retrieval issues point towards embeddings, chunking strategies or search quality. Reasoning issues point towards prompts, model behaviour or response generation.

Without observability, both problems look identical from the outside. With complete traces, the distinction becomes immediately clear, allowing us to focus our efforts on the right part of the system.

Measuring more than answer quality

Tracing every interaction has also improved how we evaluate the operational performance of MMMGPT. Each traced model call captures token consumption, response latency and cost.

Figure 4. Langfuse's Usage by Type dashboard provides visibility into input, output, cached input and total token usage, helping us monitor model usage patterns over time.

Because these requests pass through the same tracing layer, we can analyse usage by user and by session without maintaining separate tracking systems. When prompts are updated, we can immediately evaluate their impact on latency and token usage using production data rather than relying solely on anecdotal feedback.

Tracing also makes failures visible. Rather than waiting for a user to report a problem, we can see error-level events as they occur and investigate the specific traces behind them.

Figure 5. Langfuse's Observations by Level view tracks execution events and surfaces error-level events across model interactions.

Observability did not make MMMGPT smarter overnight. It made every iteration measurable.

A lightweight implementation

One of the reasons we adopted Langfuse was its simplicity. The integration required only a few environment variables and minimal code changes. There was no need for additional infrastructure or significant architectural changes.

Tracing also runs asynchronously, allowing responses to be delivered to users before telemetry is sent in the background. As a result, observability adds valuable visibility without affecting the user experience.

Measurement is part of the product

At Aryma Labs, we believe measurement should not stop at marketing performance. The same discipline should extend to the AI systems we build. Every recommendation generated by MMMGPT should be explainable, measurable and continuously improvable.

Much of today's conversation around AI focuses on models, benchmarks and architectures. Those are important discussions, but they overlook a more practical question.

Can you explain exactly what your system did for a user yesterday afternoon?

If the answer is yes, improvement becomes systematic rather than speculative. Teams can identify issues faster, validate changes with confidence and continuously improve the product using evidence rather than assumptions.

That is why we believe observability is not an optional layer added after deployment. It is a fundamental capability of any production-grade AI application.

---

Thanks for reading.

For help with MMM, Causal Marketing Experiments and Experimentation, get in touch with us.

We also build some pretty cool AI products to aid Marketing Measurements. Check out our products page to know more - https://www.aryma.ai/