Business fit6.6/10
Implementation readiness6.2/10
Verdict
Testing. Useful baseline; engagement-specific evaluators still need to be written and owned.
What was tested
Baseline evaluation metrics for a retrieval-augmented pipeline against a curated question set.
Test environment
Self-hosted evaluation harness, Langfuse for trace persistence, hybrid retrieval over Qdrant.
Strengths
- Sensible default metrics for retrieval and generation quality
- Reasonable starting point before authoring engagement-specific evaluators
- Composable with team-owned datasets and traces
Limitations
- Default metrics rarely match the actual product question
- Metric stability depends on judge model selection
- Limited visibility into why a score moved between runs
Risks
- Treating off-the-shelf metrics as a substitute for engagement-specific evaluators
- Optimising the score rather than the underlying product outcome
Integration notes
We use Ragas as a baseline layer and write engagement-specific evaluators on top. Results are persisted alongside traces so decisions remain auditable.