LangFuse

Langfuse is an open-source LLM engineering platform that gives developers full visibility into how their AI applications behave in production. It combines tracing, prompt management, and evaluation into one connected workflow, so you can move from prototype to production without flying blind. For AI engineers building agents, RAG pipelines, or any system that calls an LLM, Langfuse is the observability layer that tells you what is actually happening inside your app.

How Langfuse works

When an LLM application runs, it executes a chain of steps: retrieving context, calling a model, running a tool, returning a response. Without instrumentation, you only see the final output. Langfuse captures every step as a trace, a structured record of that entire execution. Each trace is made up of nested spans, one per step, so you can inspect exactly where latency is building up or where a bad output originates.

Here is how the core mechanism works:

• Instrumentation: You add a Langfuse SDK (Python or JavaScript) or connect via OpenTelemetry to your existing setup. Drop-in wrappers for OpenAI, LangChain, LlamaIndex, and LiteLLM mean you often need just one line of code to start collecting traces.
• Trace ingestion: Every LLM call, tool invocation, retrieval step, and API request gets logged as a span. Spans nest hierarchically, so a multi-step agent appears as a tree you can expand and inspect.
• Prompt management: Prompts live in Langfuse rather than hardcoded in your repo. You version them, deploy them via label (production, staging, dev), and update them without a redeploy. The platform links each prompt version to the traces it produced.
• Evaluation: Langfuse scores your outputs using LLM-as-a-judge, heuristic functions, manual annotation, or user feedback. Scores attach to specific traces, so you can filter by quality, spot failure patterns, and compare prompt versions with real metrics.
• Datasets and experiments: You build test sets from real production traces, then run experiments to compare how prompt changes or model swaps affect quality before you ship.

What you can build with Langfuse

Langfuse suits any developer who is moving an LLM application beyond a one-off script and into something real users depend on.

• RAG (Retrieval-Augmented Generation) pipeline monitor: A system that retrieves documents from a vector database before generating an answer. Langfuse traces each retrieval call alongside the LLM call, so you can see whether the right documents are being fetched and whether they are actually improving the response.
• Multi-agent debugger: An agentic workflow where one orchestrator calls subagents, each calling tools. Langfuse renders the full agent graph visually, making it possible to see which agent is slow, which tool is failing, and where the execution diverges from expected behavior.
• Prompt iteration system: A team workflow where product, engineering, and QA can all propose, test, and deploy prompt changes from the Langfuse UI without touching application code. Version history, metrics per version, and rollback all come included.
• Cost and latency dashboard: A production monitor that tracks token usage and inference cost by user, session, or model. Useful when you are running multiple model providers or need to attribute costs to specific features or customers.
• LLM evaluation pipeline in CI/CD: A test suite that runs your curated dataset through the latest prompt version on every pull request and flags regressions before they reach production.
• Hallucination detection layer: An evaluation setup using LLM-as-a-judge that automatically scores every production trace for factual correctness and flags suspicious outputs for human review.