Validated in our current evaluations for tracing, debugging, and reviewing LLM or agent runs where observability, prompt/version review, and evaluation workflows matter. It remains one part of an evaluation strategy, not the strategy by itself.
End-to-end tracing of a multi-step agent workflow and a RAG pipeline, prompt and version visibility, score capture from custom evaluators, dataset-driven offline evaluation runs, and per-trace cost and latency attribution across model providers.
Self-hosted Langfuse with Postgres persistence, traces emitted from a TypeScript agent service and a Python retrieval service behind an internal model gateway, custom evaluators writing scores via the SDK.
- Trace model maps cleanly to agent steps, tool calls, and retrieval spans
- Prompt and version tracking integrates with scoring and dataset runs
- Self-hosting and Postgres backend keep traces inside the engagement boundary
- Cost, latency, and score views support concrete review rather than vibe checks
- Does not replace an evaluation strategy — custom metrics still have to be designed and written
- Quality of every view depends on the quality of upstream instrumentation
- Sampling, retention, and PII handling require intentional configuration before any sensitive workload
- Dataset and eval workflows are still maturing for complex multi-stage suites
- UI density grows quickly under high-traffic apps and needs filtering discipline
- Treating dashboards as a substitute for owning evaluation logic and acceptance criteria
- Sending raw prompts and outputs without a deliberate redaction and retention policy
- Drifting trace retention below what audit review or incident reconstruction requires
- Adopting it widely without budgeting time to maintain instrumentation as systems change
Sits as the tracing and evaluation-review layer beneath agent and retrieval services in a governed AI architecture: spans from orchestration and retrieval flow in, custom evaluators write scores against datasets, and the same surface is used for prompt versioning and cost review. Evaluation logic itself stays in code we own.