My Practical Experience with RAG: How Documents Become Reliable AI Answers

In the context of my continuing education in Applied AI & Data Science at MIT, I implemented a complete Retrieval-Augmented-Generation pipeline, or RAG for short, in Python. For me, it wasn't just about connecting a Large Language Model with a document. I wanted to understand how the individual technical building blocks interact and how to generate answers that are traceable, limited, and as closely aligned as possible with the available sources.

Out of consideration for the protected course content, I will not describe the specific use case, the documents used, the tasks, or the complete prompts here. Instead, the article focuses on the general technical workflow and my personal insights from the implementation.

Why RAG is Necessary at All

A Large Language Model possesses broad general knowledge but doesn't automatically know a company's internal documents. It can also fabricate information or make outdated statements if it lacks the necessary foundation.

RAG therefore augments the language model with an upstream search process:

1. Relevant content is retrieved from a knowledge source.
2. Only this content is passed to the model as context.
3. The model formulates an answer based on this.

The language model is not retrained in this process. Instead, it receives precisely the information relevant to a specific question at runtime.

For me, this was an important realization: A RAG system is not simply a chatbot with document upload. It is a pipeline consisting of document processing, semantic search, context building, prompting, and controlled text generation.

1. Setting up the Technical Environment

The implementation was done in Python. Several specialized components were combined for document processing, tokenization, embedding creation, and vector search.

The central components were:

• pypdf for reading PDF documents,
• LangChain for document structure and text splitting,
• tiktoken for token-based chunking,
• Gemini models for embeddings and text generation,
• ChromaDB as a persistent vector database,
• an OpenAI-compatible client for model calls.

This setup showed me that a RAG solution usually doesn't consist of a single framework. Rather, quality arises from the clean interplay of several components.

2. Reading in and Unifying Documents

The pipeline was structured to process both a single PDF and a directory containing multiple PDF files.

For individual files, text was extracted page by page and then merged. For multiple files, documents could be loaded via a Directory Loader.

The result of this phase was not yet a searchable knowledge base. Initially, only extracted, unstructured text was available.

At this point, it became clear how important the quality of the source documents is. If a PDF cannot be read cleanly, headings, tables, or text contained within images may be completely missing. A RAG system can only find information that has been correctly extracted and indexed beforehand.

3. Token-Based Chunking with Overlap

A complete document usually cannot be meaningfully passed as a single context to the language model. It would be too large, too expensive, and too imprecise for semantic search.

Therefore, the text was broken down into smaller sections, called chunks.

In the notebook, a recursive text splitter was used that not only counted characters or words but oriented itself on the actual tokenization of a language model. The maximum chunk size and the overlap between adjacent sections were explicitly configured.

Overlap is important because relevant information often starts at a chunk boundary and continues into the next section. Without overlap, the semantic connection could be broken.

At the same time, this creates a conflict of objectives:

• Chunks that are too small may lose important contexts.
• Chunks that are too large contain a lot of irrelevant text.
• Excessive overlaps create unnecessary duplicates and increase costs.
• Insufficient overlaps can separate related content.

Chunking is therefore not purely a technical preprocessing step, but one of the most important quality decisions in a RAG system.

4. Semantic Representation through Embeddings

After chunking, the individual text sections were converted into embeddings.

An embedding is a numerical representation of the semantic content of a text. Texts that are semantically similar should be as close as possible in the vector space, even if they don't use the same words.

The embeddings were generated not individually, but in batches. This reduces the number of API calls and makes processing larger document volumes more efficient.

Each chunk also received a unique ID. This allowed later tracing of which sections were used for an answer.

What was particularly interesting to me was that a RAG search doesn't primarily look for identical terms. A user's question can be formulated differently than the document and still find the appropriate section if both are semantically similar.

5. Storage in a Persistent Chroma Database

The generated text sections and embeddings were stored in a Chroma vector database.

The collection was configured with Cosine Similarity. This similarity measure evaluates the direction of two vectors and is commonly used to identify semantically similar texts.

The database was stored persistently. This meant the entire embedding process didn't have to be executed again with every restart.

This has two immediate advantages:

• shorter startup and processing times,
• lower costs due to avoided embedding calls.

In a productive system, a controlled strategy for updates would also need to be implemented. If documents are changed, it must be clear which chunks need to be newly generated, updated, or deleted. In the notebook, the initial focus was on building and understanding the complete pipeline.

6. Embedding the User Question as Well

Once the knowledge base was prepared, a user question could be processed.

For this, the question was translated into a vector using the same embedding model. This query vector was then compared with the stored document vectors.

The vector database returned the best-matching text sections. In the notebook, a limited number of the most relevant chunks were retrieved.

Among other things, the following were returned:

• the chunk IDs,
• the associated texts,
• possible metadata,
• the calculated distances.

This concluded the retrieval phase. At this point, the language model had not yet generated an answer. The pipeline had only determined the most likely relevant context.

7. Building the Context in a Controlled Manner

The found chunks were then assembled into a common context.

Each section received its ID as a visible header. The chunks were clearly separated from each other so that their boundaries remained recognizable in the prompt.

This structure had two functions:

1. The model could better distinguish between the provided information.
2. The sources used could later be returned based on the chunk IDs.

The complete prompt consisted of a system instruction, the retrieved context, and the actual user question. Context and question were separated by clearly defined markers.

This separation is important because document content should not be accidentally interpreted as a system instruction. In productive environments, this area would also need to be secured against Prompt Injection and manipulated document content.

8. Deliberately Limiting Hallucinations

The model received a clear instruction to answer exclusively based on the provided context.

If the requested information was not contained in the context, it should not invent a plausible answer, but explicitly state that the answer is unknown.

Furthermore, the answers were deliberately requested to be concise and source-related. A deterministic setting was used for generation, and the maximum output length was limited.

These measures can reduce hallucinations:

• clearly delimited context,
• strict grounding instruction,
• defined fallback answer,
• low or deterministic temperature,
• limited answer length,
• return of used sources.

However, hallucinations cannot be completely prevented by these means. An LLM can misinterpret an existing text or incorrectly combine different sections. Therefore, evaluation remains a necessary component of any serious RAG solution.

9. Returning Answer and Sources Together

The final function combined retrieval and generation in a single workflow:

1. Receive question,
2. Create query embedding,
3. Retrieve relevant chunks,
4. Generate context,
5. Assemble prompt,
6. Generate answer,
7. Return used chunk IDs.

As a result, not only the answer text but also the used sources and the complete retrieval details were output.

I find this separation particularly important. A pure AI answer is difficult to verify. If, however, it becomes visible which document sections it is based on, users or downstream processes can trace the statement.

Source Attribution is thus not just a convenience feature, but an essential component of trust, governance, and quality assurance.

What I Learned from the Implementation

Through the notebook, I understood that the quality of a RAG system is determined by several interdependent factors:

• the quality of the source documents,
• the extraction of the text,
• the chunk size and overlap,
• the embedding model used,
• the retrieval strategy,
• the number of retrieved chunks,
• the construction of the context,
• the clarity of the prompt,
• the generation parameters,
• the evaluation of the answers.

A powerful language model cannot reliably compensate for a weak retrieval pipeline. If the wrong context is found, even a linguistically perfect answer can be factually incorrect.

Conversely, a good semantic search alone is also not sufficient. The found content must be passed to the model in such a way that it uses it correctly and limits its answer to it.

What I Would Add for Productive Further Development

The notebook implementation conveyed the complete technical core of a RAG pipeline. For a productive system, I would supplement this approach with additional components:

• structured metadata and metadata filters,
• re-ranking of the found chunks,
• automated evaluation datasets,
• measurement of Retrieval Precision and Faithfulness,
• document and index versioning,
• role and permission checking,
• protection against Prompt Injection,
• caching of recurring questions,
• logging, tracing, and cost monitoring,
• defined processes for updating the knowledge base,
• automated regression tests for typical questions.

My background in software quality and test automation particularly influences my view on RAG. An answer that seems convincing in a demonstration is not yet a robust AI system. Only repeatable tests, traceable sources, controlled error cases, and monitoring turn it into a long-term usable solution.

My Conclusion

My practical work with RAG showed me how unstructured documents can be transformed into a semantically searchable knowledge base. I implemented the complete workflow myself: from reading in and token-based chunking, through embeddings and persistent vector storage, to semantic search, controlled prompt construction, and the output of source-based answers.

It was particularly valuable for me to learn RAG not as a ready-made library function, but as a traceable architecture. This allows me to consciously evaluate, exchange, and optimize the individual components.

The central insight for me is:

A good RAG system should not answer as much as possible. It should find the right information, limit its answer to this basis, and transparently show where it comes from.

The article deliberately abstracts from the protected case study content and exclusively describes the general technical workflow and my insights gained from it.