Your RAG application can return a similarity score of 0.92 and still give the user the wrong document. Embeddings make semantic search possible, but three common mistakes can quietly break retrieval before the language model even generates an answer.
If you are building AI applications with Python, OpenAI embeddings, or PostgreSQL with pgvector, understanding these mistakes can save hours of debugging.
1. Mistake One: Splitting Text by Character Count Instead of Meaning
Before creating embeddings, most RAG systems divide documents into smaller pieces called chunks. Each chunk is converted into a vector so the application can retrieve relevant passages when a user asks a question.
A common beginner approach is to split every 1,000 characters, regardless of where sentences, paragraphs, or sections end. This can separate a question from its answer or divide an important condition from the rule it qualifies.
Consider this policy sentence: “Employees can request reimbursement for travel expenses up to $500, provided the trip receives manager approval before booking.” If a character-based split separates “provided the trip receives manager approval before booking” from the reimbursement limit, retrieval may return the $500 amount without the approval condition.
The language model may then produce an incomplete answer even though the original document contains the correct information. The problem is not necessarily the model; the retrieval system supplied incomplete context.
Start by splitting at natural boundaries such as paragraphs, headings, and sentences. A practical experiment is to compare chunks of roughly 300–600 tokens with a small overlap, such as 50–100 tokens, then measure whether the correct evidence appears in the retrieved results.
These numbers are starting points, not universal settings. Technical manuals, legal documents, code, and FAQs may require different chunk sizes and strategies.
2. Mistake Two: Mixing Embedding Models
An embedding model converts text into a list of numbers representing aspects of its meaning. A vector database compares these representations to find passages that are semantically similar to a user's question.
The problem begins when you index documents with one embedding model and search them with another. Different models can produce vectors with different dimensions and different learned representations, so their vectors are not automatically compatible.
For example, OpenAI's text-embedding-3-small produces 1,536 dimensions by default, while text-embedding-3-large produces 3,072. If your PostgreSQL column expects 1,536 dimensions, inserting a default 3,072-dimensional vector will fail unless you deliberately configure compatible dimensions.
Even when two models produce vectors with the same number of dimensions, that does not mean their vectors share the same semantic space. Comparing them directly can produce meaningless rankings.
Keep track of the embedding model, dimensions, preprocessing rules, and version used for each index. If you change models, plan a controlled re-embedding process and rebuild or migrate the affected vectors instead of silently mixing old and new representations.
3. Mistake Three: Trusting Cosine Similarity Without Inspecting Results
Cosine similarity measures the angle between two vectors and is commonly used to rank semantically similar text. It is useful, but a high score does not prove that a passage answers the user's question.
Imagine a user asks, “How many days do I have to return a laptop?” Your search returns passages about the return policy, laptop warranty, shipping deadlines, and refund processing. A passage about warranty coverage might score highly because it shares similar language, yet it does not answer the question.
Never evaluate retrieval using similarity scores alone. Inspect the actual text, its source document, and whether it contains the evidence needed to answer the question.
Build a small evaluation set of real questions with known supporting passages. For each question, record the top five retrieved chunks and check whether the correct evidence appears among them; then measure metrics such as Recall@5 and the quality of the final answer.
Also check metadata filters, document permissions, duplicate chunks, and outdated content. A search system can rank a passage correctly while still returning the wrong version of a policy or information the user should not access.
4. A Practical Checklist for Debugging Retrieval
When a RAG application gives a wrong or unsupported answer, investigate the retrieval pipeline before immediately changing the language model.
- Inspect the source: Confirm that the original document contains the correct answer and that the latest version has been indexed.
- Inspect chunk boundaries: Check whether the supporting sentence or condition was split across chunks.
- Verify embedding consistency: Confirm that indexing and querying use compatible models, dimensions, and preprocessing.
- Print the top five results: Log the retrieved text, source IDs, metadata, and similarity scores during development.
- Check retrieval quality: Test known questions and measure whether the required evidence appears in the top results.
- Review filters and permissions: Confirm that access rules, document dates, and metadata constraints are applied correctly.
- Separate retrieval from generation: If the correct evidence is retrieved but the answer is still wrong, investigate the prompt, context formatting, and model response.
You can run these checks with Python, your embedding provider's API, and PostgreSQL with pgvector. Log enough information to reproduce failures, but avoid recording sensitive customer data unnecessarily.
5. Improve One Variable at a Time
When debugging, change one part of the pipeline at a time. Compare chunk sizes, embedding configurations, retrieval settings, and prompts against the same set of test questions.
For example, test 100 representative questions before and after changing chunking. Record how often the correct evidence appears in the top five results, how often the final answer is supported by that evidence, and how long each request takes.
Embeddings are a foundation of many RAG applications, but they are not a guarantee of accurate answers. Reliable retrieval comes from good document preparation, compatible models, careful evaluation, and inspecting what the system actually returns.
Which of these three have you already made?
Join the KKRGENAI community discussion to share your retrieval bugs, debugging methods, and lessons from building RAG applications.
