Retrieval is a data problem before it is a model problem
Better answers usually start with better document structure, metadata, chunking, retrieval tests, and failure analysis — not only a larger model or a different vector database.
Track lead — Retrieval & Knowledge Intelligence →Why this note exists
This note captures a technical observation from Yuauri's AI engineering work: tool evaluation, implementation planning, risk review, architecture decisions, or execution patterns. It may later become a full tool evaluation, implementation playbook, or internal delivery pattern.
Most retrieval failures attributed to "the model" or "the vector database" turn out, on inspection, to be data and architecture problems. The corpus is inconsistent. Documents have unclear ownership and unclear freshness. Chunking is uniform where it should be structural. Metadata is sparse or absent. The system has never been tested against the questions users actually ask.
Swapping the embedding model or the vector database rarely fixes any of that. It changes one variable in a system whose dominant failure modes live elsewhere.
The Yuauri view is that retrieval quality is mostly determined upstream of the model. The questions that decide whether a knowledge system is trustworthy are operational and editorial:
- Source data — which documents are authoritative, who owns them, how often they change, and how their structure is preserved when ingested? - Chunking — does the chunking strategy follow the natural seams of the documents (sections, clauses, procedures), or does it cut through them? - Metadata — are source, version, owner, jurisdiction, sensitivity, and date attached to each chunk so that filtering, ranking, and citation are possible? - Query strategy — are queries rewritten, expanded, or routed before retrieval, and is hybrid (keyword + vector) used where it actually helps? - Coverage — for a representative set of real questions, does the system retrieve the documents that should answer them, and does it avoid the ones that should not? - Failure cases — what does the system do with ambiguous, out-of-scope, stale, or conflicting sources?
Vector databases like Qdrant and frameworks like LlamaIndex are useful infrastructure inside this architecture. Qdrant gives a serious vector store with payload filtering and operational characteristics that scale. LlamaIndex gives composable patterns for ingestion, indexing, and query construction. Neither, on its own, produces a trustworthy knowledge system. They are components inside one.
Retrieval quality has to be measured, not assumed. That means a representative evaluation set drawn from real usage, expected source coverage for each question, explicit failure cases, and regression checks that run when the corpus, chunking, metadata, embedding model, or retrieval strategy changes. Without that, "we improved retrieval" is an opinion.
When Yuauri evaluates tools for a knowledge system, a recommendation is incomplete if it only names a vector database and an embedding model. It also has to describe data readiness, the retrieval strategy, the evaluation baseline, and who in the organization is responsible for keeping the corpus and its metadata correct over time.