Ask most developers what RAG is and you'll hear "you put your documents in a vector database." That's not wrong, exactly. It's just answering a different question. It's like asking what a web app is and being told "you put your data in Postgres." True, often. But it skips the part that actually matters, and it quietly turns one implementation detail into the definition.
So here's the version I wish someone had given me when I started, aimed squarely at people who build ordinary CRUD apps for a living.
The model has amnesia, and RAG is how you brief it
A language model knows nothing about your documents. It wasn't trained on your company wiki, your Obsidian vault, or last week's incident report. You can't fix that by asking nicely. The model has no way to go look things up.
RAG — retrieval-augmented generation — is the unglamorous fix: before you ask the model a question, you go find the relevant material yourself and paste it into the prompt. The model then answers using text you handed it, not memory it doesn't have. That's the whole idea. Retrieve the right context, then generate an answer from it.
Strip away the jargon and the shape is something you already build every day:
question -> find the relevant stuff -> put it in the prompt -> answer
The middle step is the entire game. The model is mostly along for the ride.
It's a WHERE clause that runs before the model
If you've written a search endpoint, you've already built the hard part of RAG. Think about how you render a page of results: a user types something, you run a query that selects the rows that match, and you show those rows. You don't hand the database all ten million rows and hope. You filter first.
RAG is the same move, with one extra step at the end:
-- the part you already know how to do
SELECT content FROM docs WHERE <relevant to the question>
...and then you take those few rows, drop them into a prompt, and let the model write prose over them. The retrieval is a WHERE clause. The model is the SELECT ... formatting layer that turns rows into a sentence with citations.
Once you see it this way, the real question stops being "which vector database" and becomes the one that was always underneath: how do I find the relevant stuff? That's a search problem. You've solved search problems before.
Where embeddings come in — and where they don't
Embeddings are one answer to "how do I find the relevant stuff," and they're a genuinely good one. An embedding turns a chunk of text into a list of numbers positioned so that similar meanings land near each other. Store those vectors, embed the question the same way, and grab the nearest neighbors. Now "how do I rotate my credentials" can match a doc titled "key rotation policy" even though they share no words.
That fuzzy, meaning-based matching is the thing keyword search can't do, and it's why embeddings got popular. Worth having in the toolbox.
But somewhere along the way "RAG" and "stuff a vector database" became synonyms, and that's the part I'd push back on. Embeddings are a retrieval strategy, not the definition of RAG. They come with real costs that nobody mentions in the quickstart:
- You have to chop your documents into chunks, and the model only ever sees the chunk, not the document it came from. Pick the chunk size wrong and you either slice sentences in half or bury the signal in noise. A shocking amount of "the AI gave a dumb answer" traces back to a bad chunk, not a bad model.
- Structure gets flattened. Your knowledge already has shape — folders, headings, a doc that links to three others. Embed everything into a soup of vectors and that shape is gone. You traded the table of contents for a pile of index cards.
- It's a whole new piece of infrastructure to run, populate, and keep in sync with the source of truth.
Sometimes that trade is worth it. Often, for a tidy, well-organized corpus, it's solving a problem you didn't have.
At least three ways to find the relevant stuff
If retrieval is just "find the relevant stuff," then embeddings are one option among several, and you can mix them:
- Keyword search. Plain full-text search — the
bm25ranking your database already ships with. Unbeatable when the question and the docs share vocabulary, which is more often than vector-search enthusiasm admits. Cheap, instant, no extra infrastructure. - Vector search. The embeddings story above. Earns its keep on fuzzy, synonym-heavy questions over messy text.
- Structure-aware retrieval. Don't flatten the documents — navigate them. If your corpus has a hierarchy, you can let the model read the table of contents and descend toward the answer the way a person flips to the right chapter, instead of guessing at chunks. The structure becomes the index.
That third one is the approach I've been building a tool around, because most of the corpora I actually care about — notes, docs, a vault — already come with structure that vector search throws away. More on that build in the posts that follow this one.
The thing to actually internalize
The model cannot rescue bad retrieval. If the right document never makes it into the prompt, no amount of prompt-engineering or a fancier model conjures it back — the information simply isn't in the room. People burn weeks tuning the generation half (better prompts, bigger model, more examples) when their problem lives entirely in the retrieval half.
So when you sit down to build RAG, don't start by shopping for a vector database. Start with the question you already know how to ask: given this question, how do I reliably find the handful of documents that answer it? Answer that well and the AI part is almost an afterthought. Answer it badly and the best model in the world will confidently tell you something wrong.
RAG isn't a database you buy. It's a search problem you solve, with a language model stapled to the end.