AI Agent Observability in Production: How to Trace an Enterprise Agent with Self-Hosted Langfuse

What AI agent observability is

AI agent observability is the practice of capturing every step an agent takes, the calls to the model, the tools it invokes, the context it retrieves and the control-flow decisions it makes, as structured traces that can be inspected, filtered and evaluated.

The definition is easy. What it implies in production is not. An agent running in an enterprise answers real users, on real data, inside processes where someone will eventually ask why a given answer was what it was. At that point “the model said so” is not an answer. You need to be able to open any single turn, weeks later, and see what the agent was given, what it decided at each step, what it executed and what came back.

That’s also why classic logging and application monitoring aren’t enough on their own. Logs record events that the code was written to emit. Metrics like latency, error rate and throughput tell you whether the service is healthy. Neither tells you why the model picked one tool over another, which documents were in its context when it did, or that it went around the loop three times before settling on an answer. In an agentic system the most important decisions aren’t written in the code. They are made at runtime by the model, and they have to be recorded at runtime too.

The vocabulary: traces, observations, sessions

Agent observability tools share a small vocabulary, and it’s worth fixing it before going further.

• A trace is one unit of work, end to end. For a conversational agent, the natural unit is the turn: one user message, everything the system does about it, one answer.
• An observation is a step inside a trace. Observations nest: a step of the agent contains the model call it made, a tool execution contains the query it ran. Some observations are plain spans, a unit of work with a start, an end, an input and an output. Others are generations, calls to a model, which carry the prompt, the completion, the token usage and the latency.
• A session groups the traces of the same conversation, in order. An agent’s answer at the fifth turn often depends on what was said at the second, so a turn read in isolation can be misleading.
• Attributes such as the user, the environment or tags make traces findable: they’re what you filter on when you’re looking for a pattern rather than a single turn.

The structure matters more than any single field. A good trace reads like the execution itself: you open it, and the nesting of its observations tells you what happened, in what order, and inside what.

Why agentic components need dedicated observability

Any software in production needs monitoring. Software with an agentic component needs something more specific, for three reasons.

Failures hide in the middle. A traditional service tends to fail loudly: an exception, a 500, a timeout. An agent can fail quietly. It can choose the wrong tool, retrieve the wrong context, invent a field that doesn’t exist, retry twice, and still produce a final answer that looks perfectly plausible. The error isn’t in the output. It’s three steps before it, and you only see it if those three steps were recorded.

The same input doesn’t give the same run. Models are not deterministic, and above all, an agent’s behaviour depends on everything that was in its context at that moment: the system prompt, the documents a retriever returned, the conversation so far, the results of earlier tool calls. Two runs with the same question can diverge only because the context was different. Without the context captured as it was, a failure can’t be reproduced, and what can’t be reproduced can’t be fixed.

The model decides the control flow. In ordinary software, the path through the program is written in the code, and reading the code tells you what will happen. In an agent, the code defines the options and the model chooses among them: which tool to call, with which arguments, when to stop. Those choices are the program’s control flow, and they exist only at runtime. Observing an agent means recording its decisions with the same care you’d record its data.

How modern agents are built, and what tracing them means

At its core, a modern agent is a model calling tools in a loop. Everything that makes it work in production, though, lives in the code around the model: the harness. To trace an agent properly you have to trace the harness as much as the model.

Orchestration frameworks like LangGraph make the harness explicit. Instead of a while loop buried in a function, the agent is a graph: named nodes for each step, and edges, some of them conditional, for the decisions about where to go next. That’s good for the code, and just as good for observability. When every step has a name and a boundary, every step can become an observation in the trace, and the shape of the trace follows the shape of the graph.

The loop and its harness. The harness assembles the context, sends it to the model, executes the tool calls the model asks for, feeds the results back and decides when the loop stops: because the model produced an answer, or because a budget of iterations ran out. Tracing the loop means seeing every iteration as its own step, in order, and seeing why it ended. “The agent answered after four iterations” and “the agent hit its iteration limit after four” look the same from the outside and mean very different things.

Tool calling. Tools are how an agent acts: searching, querying, calling an API. At each step the model sees a set of available tools and picks one, with arguments it writes itself. Tracing a tool call means recording which tool was chosen, with which arguments, and what came back, including when what came back was an error. Most agent bugs show up here first: the right tool with the wrong arguments, or the wrong tool for the question.

Context engineering. Everything the model knows at a given step is in its context window: the system prompt with its instructions, the context a retriever selected, the history of the session, the results of previous tool calls. That window is where every decision is made, so it’s the single most important thing to capture. Not a summary of it, the full prompt as the model received it. When an agent does something strange, the explanation is almost always in there: a document that shouldn’t have been retrieved, an instruction lost in a long history, a tool result the model misread.

Deterministic guardrails. In a production agent not everything is left to the model. The harness contains plain code that checks what the model produces before it has effects: validating a query, filtering an action, enforcing a permission. A guardrail that refuses something is itself a decision point, and it belongs in the trace with its verdict and its reason. Otherwise a refused action and an action that never happened look the same.

A real case: an agentic text-to-SQL system on-premise

In my day-to-day work I build software products with an agentic core. That distinction matters: an agentic system in production is software first, with everything that comes with it, versions, deployments, users, support, and an agent at the centre of it. One of these products is a text-to-SQL system that lets business users query an enterprise ERP in natural language. It’s deployed on-premise, inside the customer’s infrastructure, next to the database it queries.

Its agentic core follows the shape described above:

• A user asks a question in natural language.
• Before the model writes anything, a retriever selects the relevant part of the ERP schema: table definitions and a few example queries.
• An agent built with LangGraph works in a loop with two tools. retrieve_context asks the retriever for more schema when what the model has isn’t enough. execute_sql takes the SQL query the model wants to run.
• Every query goes through a deterministic guard before it reaches the database. A refused query goes back to the model as feedback, and the loop continues.
• When a query runs successfully, the rows are the answer, and the turn ends.

For a system like this, observability is not optional. A business user who gets a number from it needs to be able to trust it, and whoever supports the product needs to be able to answer, for any turn, three questions: what the model saw, what it decided, and what actually ran.

Why Langfuse

The choice of the observability tool followed from the deployment model of the product, not the other way round.

The customer’s constraint becomes the tool’s constraint. Enterprise customers wanted the product on-premise for a simple reason: their data must not leave their servers. A trace of a text-to-SQL turn is full of exactly that data. The prompt contains the schema of their ERP and example queries written on it; the tool results contain rows from their database; the inputs are their users’ questions. Sending traces to a third-party cloud would have undone, in the observability layer, the guarantee the product gave in every other layer. So the observability tool had to be self-hostable, inside the same perimeter as the product. Langfuse is.

Open source, with no per-event bill. Langfuse’s core is MIT-licensed. A small set of enterprise features sits under a separate licence, but tracing, sessions, datasets and experiments are in the open-source core. Self-hosted, there’s no usage limit and no price per trace: the cost is the infrastructure it runs on. For a product deployed at several customers, each with its own installation, that’s what makes the model sustainable.

It fits the stack. The agent is built with LangGraph and talks to the model through an OpenAI-compatible client. Langfuse has an integration for both. A callback handler passed to the LangGraph run records every node of the graph with its input and output state. A drop-in wrapper for the OpenAI client records every model call as a generation: full prompt, completion, tool calls, token usage, latency. And the SDK is built on OpenTelemetry, so the traces follow a standard rather than a proprietary format. Getting a complete trace of a turn took a few lines of code, not a re-architecture.

Anatomy of a trace in Langfuse

The unit of work in the system is the turn: one question, everything the agent does about it, one answer. So the unit of tracing is the turn too, one trace per turn.

The root observation. Each turn opens a root observation of type agent that wraps everything else. Its input is the question and the context of the request. Its output is a summary of the turn: the answer, the SQL error if there was one, the number of rows returned and the number of iterations of the loop. The trace also carries the user_id and the session_id, so any turn can be read as part of the conversation it belongs to.

from langfuse import get_client, propagate_attributes
from langfuse.langchain import CallbackHandler

async def answer(question: str, user_id: str, session_id: str):
    langfuse = get_client()
    with propagate_attributes(user_id=user_id, session_id=session_id, trace_name="agent_turn"):
        with langfuse.start_as_current_observation(
            as_type="agent", name="agent_turn", input={"question": question},
        ) as root:
            state = await graph.ainvoke(
                {"question": question}, config={"callbacks": [CallbackHandler()]}
            )
            root.update(output={
                "answer": state["answer"],
                "sql_error": state["sql_error"],
                "row_count": len(state["rows"] or []),
                "iterations": state["iterations"],
            })
    return state

The graph nodes. Inside the root, the LangGraph callback records one observation per node execution, named after the node, with the state that went in and the state that came out. The graph has a small set of nodes, and each has a clear role:

• get_or_create_session loads the conversation the turn belongs to;
• init_messages assembles the prompt and runs the first retrieval of schema and examples;
• agent is one call to the model, which either requests tools or stops;
• tools executes the tool calls the model requested;
• save stores the turn and closes it.

A turn that loops shows agent and tools alternating, once per iteration. The structure of the trace is the structure of the graph.

The generations. Every call to the model is a generation nested in its agent node. This is where “what the model saw” lives: the full prompt, including the system instructions, the schema and examples the retriever selected, the conversation so far and the results of earlier tool calls. Next to it, the completion and the tool calls the model wrote.

The token usage and latency on each generation answer questions that the turn’s total time can’t. In a loop the context grows at every iteration, because each tool result is appended to it, so the second and third model calls of a turn are usually larger and slower than the first. Reading them generation by generation shows where the time of a slow turn actually went: a model call with an oversized context, a long tool execution, or simply one iteration too many.

One span per query attempt. In a text-to-SQL system the most important output of the model isn’t its text, it’s the query. So each attempt to run SQL gets its own explicit span: the SQL as input, and as output the row count, the error, and whether it succeeded. A query refused by the guard and a query that failed in the database both land here, with the reason.

from langfuse import get_client

def trace_sql_attempt(sql: str, row_count: int, error: str | None) -> None:
    with get_client().start_as_current_observation(
        as_type="span", name="generated_sql", input={"sql": sql},
    ) as span:
        span.update(output={"row_count": row_count, "sql_error": error, "success": error is None})

Reading and debugging traces

A trace is read from the outside in. First the root: did the turn end with an answer, how many rows, how many iterations? Then the shape of the loop: which nodes ran, how many times, in what order. Only then the detail: the generation that shows what the model saw, the span that shows what actually ran. Two examples, reconstructed on an illustrative schema, show how this works in practice.

Scenario 1: an invented column

A user asks which customers ordered more than 10,000 in the third quarter. The answer is right, but the root shows two iterations where one would be expected.

Reading it from the outside in:

1. The root says the turn ended well, with 42 rows, in two iterations.
2. The nodes show the loop: agent, tools, agent, tools, then straight to save. Two attempts at a query, the second successful. After a successful query the graph goes directly to save without calling the model again: the rows are the answer.
3. The first generation shows the full prompt. The schema the model received has a total column on orders and nothing called order_value. The model wrote order_value anyway: a name that looks right and doesn’t exist.
4. The first generated_sql span shows the guard refusing the query, with the reason. Nothing reached the database.
5. The second generation has the same prompt plus the guard’s feedback, and a query that uses total. The second span shows it ran and returned 42 rows.

Without the trace, this turn is just “slower than usual”. With it, it’s a precise finding: the model invents a plausible column name when the real one is less obvious, and the guard catches it at the cost of one extra round trip. That’s something a schema description or an example query can fix, and the next traces will show whether it worked.

Scenario 2: context that wasn’t enough

A user asks for the average delivery delay per carrier. The first retrieval brings in the schema of orders and customers, but nothing about shipments. Here the useful view is the agent graph, which Langfuse draws from the LangGraph nodes of the trace: one node per step, with a counter for how many times it ran.

The graph tells the story at a glance: two passes through the loop. Opening the steps confirms what each pass did:

1. The first generation shows a prompt with the schema of orders and customers only. The model doesn’t guess a shipments table. It calls retrieve_context, asking for shipment and carrier data.
2. The first tools step shows the retrieval: the definitions of the shipments and carriers tables come back and are added to the context.
3. The second generation shows a prompt that now contains those tables, and a call to execute_sql with a query that joins them.
4. The second tools step holds the generated_sql span of a query that ran, and the turn goes to save.

This is the behaviour you want: when the context isn’t enough, the agent asks for more instead of inventing it. The trace is what lets you verify that it does. And if the first retrieval misses shipment data for this kind of question every time, that’s a finding about the retriever, not about the model, and the trace is where you’d see it.

Reading traces in bulk

Single traces explain single turns. The same structure also answers questions across many turns, with filters instead of investigations. Which turns needed more than one iteration this week? Which generated_sql spans have success: false, and what were the reasons? Which conversations, by session_id, ended without rows? Each answer comes with the traces that justify it, one click away.

That’s the real point of agent observability in production. An agent will never be fully predictable, and it doesn’t need to be. What it needs is for every one of its decisions to be visible after the fact: what it was given, what it chose, what ran, and what came back. With that, an agent stops being a black box in the middle of the product and becomes one more component you can debug, explain and improve.