A practical guide to chunking, embeddings, and retrieval

Not every poor answer in a RAG application is the model fault. Very often, the problem starts earlier: poorly prepared documents, chunks that break meaning, weak embeddings, missing filters, or retrieval that brings back context that looks related but is not enough.

Context

In the first guide of the series, the goal was to organize the mental model for RAG: retrieve external context, give that context to the model, and validate whether the answer improved.

That flow looks simple when drawn in one line, but in practice a large part of the quality of a RAG system is decided before the model writes any answer.

1question -> search -> context -> answer -> validation

If the content was poorly split, if embeddings do not help find the right passages, if search returns material that is related but insufficient, or if metadata is ignored, the model receives weak context.

And a strong model with weak context can still generate a poor answer. Sometimes worse: a poor answer that looks confident.

The problem

The central question of this guide is: why can a RAG application answer poorly even when it uses good documents and a good model?

A common response is to switch models, adjust the prompt, or send more context. These changes can help in some cases, but they do not necessarily address the source of the problem.

• Was the content split in a way that preserves meaning?
• Are the chunks too large or too small?
• Does search find truly useful passages or only semantically similar ones?
• Is there metadata to filter by version, section, date, or document type?
• Does ranking place the best context at the top?
• Does anyone validate whether these passages support the final answer?

Mental model

1documents2  -> chunks3  -> embeddings4  -> search5  -> filters and ranking6  -> retrieved context7  -> model answer

Chunking is how content is split. Embeddings are numerical representations that help compare meaning. Retrieval is the process of bringing back the most useful passages for a question.

These three parts appear together in many RAG examples, but they are not the same thing. Treating everything as semantic search oversimplifies the problem.

Chunking: splitting is not just cutting

Chunking is the process of breaking documents into smaller pieces before indexing the content. This is necessary because whole documents are often too large, mix different subjects, and make it harder to search for a specific piece of information.

But splitting content is not just cutting by size. A chunk needs to carry a minimum unit of meaning.

• Chunks that are too large can mix subjects, add noise, and fill context with unnecessary information.
• Chunks that are too small can break an explanation in the middle, remove examples, and leave the answer without enough grounding.
• Size matters, but meaning matters more.

Embeddings: similarity is not an answer

Embeddings help represent text in a way that makes it possible to compare semantic proximity. In practical terms, they help answer which passages seem most related to a question.

This is very useful, but it has limits. A semantically close passage may not answer the question. It may discuss the same topic, use similar words, or belong to the right section, while still not providing the needed instruction.

Similar does not mean sufficient. In RAG, context needs to support the answer, not only look related.

Retrieval: bringing back useful context

Retrieval should not be understood only as taking the five most similar chunks. That can be a starting point, but more careful applications usually need to think about filters, metadata, ranking, diversity, deduplication, and context limits.

• If documentation has versions, the system can filter by the correct version.
• If there are public and internal documents, the system needs to respect scope and permissions.
• If a question is about configuration, it may make sense to prioritize guides and tutorials, not loose notes.

Retrieval is less about finding anything similar and more about choosing the context that gives the model a better foundation for answering.

Practical example

Imagine a fictional product documentation base with pages about creating a user, inviting an external user, changing an access profile, removing access, and troubleshooting login issues.

The question is: "How do I change the access profile of an external user?" A fragile flow could retrieve a general explanation about external users, a passage about creating an invite, and an old page about permissions.

A better flow would retrieve the current section about changing profiles, the specific rule for external users, and the note about the permissions required to perform the action.

The difference is not only in the model. It is in base preparation, content splitting, version metadata, the correct filter, and ranking what actually matters for the question.

Common mistakes

Splitting everything by fixed size

Splitting by fixed size is simple and can be useful as a starting point. But if that rule breaks sections, lists, examples, or prerequisites, retrieval can lose meaning.

Ignoring metadata

Metadata may look like a detail, but it is often what separates a generic search from a useful one: document version, update date, content type, section, product, language, access level, and original source.

Increasing top-k without criteria

When the answer is poor, a common reaction is to increase the number of retrieved chunks. Sometimes it helps. Sometimes it only sends more noise to the model.

Not inspecting retrieved passages

If nobody inspects what retrieval brought back, it becomes difficult to know whether the problem is in the prompt, the model, chunking, data, or search.

How to validate whether it works

A practical start is to build a small table of known questions. For each question, record the user question, expected passages, retrieved passages, generated answer, a simple human evaluation, and a note about success or failure.

1Question: How do I change the access profile of an external user?2Expected context: current profile-change section + rule for external users3Retrieved context: create invite + external users + old permissions4Diagnosis: retrieval brought back a related topic, but insufficient context5Next adjustment: review version metadata and the permission section split

This kind of validation is still not a complete eval system, but it already helps move away from "it looks good." Instead of saying "the model answered badly," you begin to see where the flow broke.

When to keep it simple

Not every case needs embeddings, a vector database, and semantic search. Sometimes, a simpler solution works better.

• Keyword search.
• Traditional filters.
• A well-organized FAQ.
• A prompt with small and fixed context.
• A clearer documentation screen.
• Business logic in code.

Lessons learned

• Chunking is not just cutting text; it is preserving units of meaning.
• Chunks that are too large can mix topics and add noise.
• Chunks that are too small can lose context.
• Embeddings help find similarity, but similarity does not guarantee an answer.
• Retrieval can involve filters, metadata, ranking, and simple rules.
• More chunks do not necessarily mean better context.
• Before switching models, it is worth looking at what is being given to the model.

Limits and caveats

This guide does not try to define an ideal chunk size, choose a vector database, or compare frameworks. Those decisions depend on content type, data volume, language, update frequency, cost, latency, permissions, and expected experience.

It also does not go into mathematical details about embeddings. To use them well in a product, the first step does not need to be mastering all the theory. But it does require understanding enough not to treat similarity as truth.

Conclusion

Improving RAG does not start only in the prompt. It starts earlier: in the content that goes into the base, how it is split, the available metadata, the search that retrieves passages, and the criteria used to validate context.

A good model helps. But it does not automatically compensate for poor context.

For developers, the practical question is: what exactly am I giving the model to answer with? If the answer is "I do not know," the next step is not switching models. It is opening retrieval, looking at the chunks, and understanding whether the retrieved context truly supports the answer.