What is RAG?
RAG (retrieval-augmented generation) is a technique in which a large language model (LLM) first retrieves relevant passages from your own document collection and then uses those passages as the basis for its answer. Answers stay anchored in your company's procedures, policies and contracts rather than only in what the model learned during training, and they can carry citations so users can check them.
The term comes from a 2020 NeurIPS paper by Patrick Lewis and colleagues (Facebook AI Research, University College London, New York University). The authors noted that language models store knowledge in their parameters but struggle to access and update it precisely, and that providing provenance for their outputs remained an open problem. Their answer was to combine “parametric memory” (a sequence-to-sequence generator) with “non-parametric memory”: a dense vector index of Wikipedia accessed through a neural retriever. According to the paper, RAG models generated more specific, diverse and factual language than a parametric-only baseline.
Today “RAG” is used more broadly than the original definition: any system that retrieves documents and places them in an LLM's context to answer a question. In companies, the most common use is an internal knowledge assistant: employees ask in natural language and the system answers from the HR handbook, operating procedures, product documentation or meeting notes, pointing to the passage it relied on.
How RAG works: six steps from documents to answers
A RAG system has two flows: an ingestion flow (runs when documents change) and a question-answering flow (runs on every question). The six steps below are the common skeleton; each involves technical choices that directly affect quality.
1. Ingest
Connect data sources (shared drives, wikis, document management systems, email, line-of-business software) and extract text from PDFs, Word files, spreadsheets and scanned images. Keep metadata: document title, version, effective date, owning department and, above all, the access-control list. Dirty input becomes wrong output, so this is where data governance basics apply.
2. Chunk
Documents are split into chunks small enough to match precisely yet large enough to keep meaning. Anthropic notes that chunk size, boundaries and overlap all affect retrieval performance. A common failure is lost context: “revenue grew by 3% over the previous quarter” is useless without knowing which company and which quarter. Anthropic's Contextual Retrieval addresses this by prepending a short, chunk-specific explanation to each chunk before embedding and keyword indexing.
3. Embed
An embedding model turns each chunk into a numeric vector that represents its meaning, stored in a vector database. The user's question is embedded with the same model so similarity can be compared.
4. Retrieve
The system finds the chunks closest to the question. Semantic (vector) search is good at matching different wordings of the same idea but can miss exact matches such as contract codes or document numbers; BM25 keyword search is the opposite. Many systems therefore use hybrid search plus a reranking step. In Anthropic's experiments, combining Contextual Embeddings with Contextual BM25 reduced the top-20-chunk retrieval failure rate by 49% (5.7% to 2.9%), and adding reranking brought the reduction to 67% (1.9%). Those results are on Anthropic's datasets; measure again on yours.
5. Generate
The selected chunks go into the prompt with the question and clear instructions: answer only from the supplied documents, say “not found” when they are insufficient, and do not infer policy.
6. Cite
The answer links back to the source document and passage. Citations let users verify, help operators spot outdated documents, and are the basis for putting a person in the loop on high-risk questions (see human-in-the-loop).
RAG vs fine-tuning vs long context: which should you choose?
The three common ways to give an LLM company knowledge solve different problems. Fine-tuning changes model parameters; it suits teaching tone, format or a repeated skill, but it is not a good way to keep facts that change weekly up to date. Long context puts whole documents in the prompt: Anthropic suggests that if a knowledge base is smaller than about 200,000 tokens (roughly 500 pages), you can include it entirely in the prompt instead of building RAG. When the collection is large, changes often and has permissions, RAG is usually the more practical choice.
- They combine well: RAG for facts, light fine-tuning for a consistent voice, long context to read a full contract once retrieval has found the right one.
| Criterion | RAG | Fine-tuning | Long context |
|---|---|---|---|
| Updating knowledge | Re-index; no retraining | Requires retraining | Swap documents in the prompt |
| Source citations | Natural, per retrieved chunk | Hard; knowledge lives in weights | Possible if explicitly requested |
| Per-user permissions | Filter at retrieval time | Not suitable | Must pre-filter documents yourself |
| Collection size | Large, thousands of documents or more | Independent of collection | Limited by the context window |
| Best for | Q&A over changing, multi-source documents | Tone, format, specialised skills | Small, stable sets; reading a whole document |
Vietnamese-specific considerations
Most RAG guides are written for English. With Vietnamese documents, check four things carefully.
Word segmentation and tokenization
In Vietnamese, spaces separate syllables, not words: “quản lý” (manage) and “hợp đồng” (contract) are two-syllable words. Some Vietnamese models require pre-segmentation; VinAI states that raw, unsegmented text must go through a word segmenter (such as RDRSegmenter from VnCoreNLP) before being fed to PhoBERT. Multilingual models such as BGE-M3 take raw text. What matters is consistency: documents and questions must pass through the same preprocessing, including for the BM25 index.
Diacritics and Unicode normalisation
Users often type without tone marks or with mistakes, and documents may use precomposed or decomposed Unicode, which look identical but differ at the byte level. Normalise Unicode (for example to NFC) for both documents and queries, and include unaccented questions as a separate slice of your test set.
Mixed Vietnamese–English text
Vietnamese business documents often mix English terms (KPI, onboarding, SLA, software names) with internal abbreviations. Keyword search catches these terms exactly, while a multilingual embedding model lets a Vietnamese question find an English passage and vice versa. An internal abbreviation glossary used for query expansion often helps noticeably.
Choosing an embedding model
Do not choose a model from English leaderboards alone. Vietnamese researchers have pointed out that Vietnamese long lacked a standard retrieval benchmark; newer efforts include VN-MTEB (41 datasets across 6 task types, built by quality-controlled translation of MTEB) and Vietnamese Context Search (VCS). Options commonly considered include BGE-M3 (multilingual, 100+ languages, inputs up to 8,192 tokens, supporting dense, sparse and multi-vector retrieval); Vietnamese fine-tunes such as AITeamVN/Vietnamese_Embedding (fine-tuned from BGE-M3 on about 300,000 query/positive/negative triplets); or commercial embedding APIs. The reliable approach is to build 100–200 real questions from your own company and measure recall on your own data before deciding.
Permission-aware retrieval: respect document access rights
The biggest risk of an internal knowledge assistant is not a wrong answer but a right answer to the wrong person: an employee asks about pay policy and receives an excerpt from the detailed payroll file. If every document goes into one shared index, the LLM has no way of knowing who may see what.
The rule is to filter by permission at retrieval time, before any passage reaches the LLM:
- Store the source document's access list (users, groups, departments) as metadata on every chunk.
- At query time, identify the user through your existing sign-in system and apply the permission filter to both vector and keyword queries.
- Sync permissions on a schedule or on events: when someone changes department or leaves, the index must follow.
- Do not rely on prompts such as “do not reveal confidential information”; a prompt is not an access-control mechanism.
- Log questions, retrieved documents and answers for internal review.
Evaluation: measure retrieval and groundedness
A fluent demo proves little. Split evaluation into two layers, because failures in each need different fixes:
- Retrieval recall: for each test question, is the correct passage in the top-k results? Anthropic uses exactly this (failure rate in the top 20) to compare techniques. The Ragas library provides Context Recall and Context Precision metrics.
- Groundedness (faithfulness): is every claim in the answer supported by the retrieved documents? Per the Ragas documentation, Faithfulness is the number of claims supported by the retrieved context divided by the total number of claims, from 0 to 1.
- Refusal behaviour: when the documents do not contain the answer, does the system say “not found” instead of inventing one?
- Permissions: include questions from under-privileged accounts to confirm confidential documents are never returned.
Illustrative example: a first evaluation set
An HR team collects 150 questions employees actually asked over the past year, recording the document and passage containing the correct answer, plus 20 questions with no answer in the documents and 20 questions asked from accounts without access. Every time the team changes the embedding model, chunk size or prompt, it reruns the full set and compares metrics with the previous run. (This is an illustrative example, not data from a real project.)
Protecting personal data in a RAG system
Internal documents often contain personal data: employee records, customer details, email. Vietnam's Law on Personal Data Protection No. 91/2025/QH15 was passed by the National Assembly on 26 June 2025 and took effect on 1 January 2026, together with implementing Decree 356/2025/ND-CP (which replaces Decree 13/2023/ND-CP), so indexing, sending passages to an AI model and keeping logs should all be reviewed against it. See our overview of the 2025 Personal Data Protection Law.
Technical measures worth designing in from the start:
- Minimise: ingest only the sources the assistant actually needs; remove or mask unnecessary personal data before indexing.
- Know where data is processed and stored (vector index, model provider, logs), especially when using services hosted abroad.
- Support deletion: when a document or a person's data must be erased, chunks, embeddings and copies in logs must go too.
- Work with your legal team on a data processing impact assessment before go-live.
Rollout plan for an internal knowledge assistant
The safe path is to start narrow and measurable, then expand. This sequence fits most mid-sized and large organisations:
- Pick a narrow scope with clear value: for example HR policy Q&A, or support documentation for one product line.
- Inventory sources, owners, freshness and access rights; clean out duplicates and outdated documents.
- Build the evaluation set from real questions before writing the first line of code.
- Build a pilot with hybrid search, permissions and citations; compare two or three embedding models on the evaluation set.
- Pilot with a small user group, with a feedback channel and a reviewer for high-risk questions.
- Set up operations: document sync, permission sync, metric monitoring, and rerunning the evaluation after every change.
- Expand to other departments and, once stable, consider making the assistant a tool for AI Agents in larger workflows (see the AI Agent operating model).
How EYC can help
EYC's Applied AI R&D practice works with organisations to assess the use case, build an evaluation set on real data and develop a permission-aware RAG pilot, with data infrastructure and deployment handled together with our cloud and data team. Scope, operating commitments and handover are defined under contract. You can contact EYC to discuss your case.
Frequently asked questions
What is RAG and what does it stand for?
RAG stands for retrieval-augmented generation. The term was introduced in the paper by Lewis et al. at NeurIPS 2020.
Does RAG eliminate AI hallucinations?
No. RAG reduces the risk by grounding answers in documents and adding citations, but the model can still misread a passage or retrieve the wrong one. Measure groundedness, let the system answer “not found”, and keep a human reviewer for high-risk questions.
When should I use RAG instead of fine-tuning?
When answers depend on facts that change often, when you need source citations, or when access must vary by user. Fine-tuning is better suited to teaching tone, format or a repeated skill. The two can be combined: RAG for facts, light fine-tuning for a consistent voice.
Which embedding model works best for Vietnamese?
There is no universal answer. Multilingual models such as BGE-M3 and Vietnamese fine-tunes are worth testing; benchmarks such as VN-MTEB help shortlist, but the final choice should rest on recall measured on your own real questions.
Do small document collections need RAG?
Possibly not. Anthropic suggests that a knowledge base under about 200,000 tokens (roughly 500 pages) can go entirely into the prompt. RAG becomes necessary when the collection is larger, changes often or needs permissions.