General AI assistants know a lot about the world, but nothing about your business: your policies, products, processes or past projects. Retrieval-augmented generation, usually shortened to RAG, is the most common way to fix that. It lets an AI answer questions using your own documents, and show where each answer came from. It powers many customer support assistants and internal knowledge tools. Here's how it works in plain English.
The idea in one paragraph
When someone asks a question, a RAG system first retrieves the most relevant passages from your documents, then augments the AI's prompt with those passages, and finally the AI generates an answer based on them. It's like giving a very fast researcher the right pages from your handbook before they answer.
How a RAG system is built
| Step | What happens | Why it matters |
|---|---|---|
| 1. Collect | Gather documents: policies, FAQs, manuals, help articles | Answers can only be as good as the sources |
| 2. Clean | Remove duplicates and outdated versions | Prevents contradictory answers |
| 3. Chunk | Split documents into passages | Retrieves precise, relevant sections |
| 4. Embed | Convert passages into numerical representations of meaning | Enables search by meaning, not just keywords |
| 5. Store | Save them in a searchable index, often a vector database | Fast retrieval |
| 6. Retrieve | Find the passages most relevant to each question | The key to accuracy |
| 7. Generate | The AI writes an answer from those passages, with citations | Grounded, checkable answers |
Where RAG is useful
- Customer support assistants answering from help articles and policies
- Internal knowledge assistants for HR policies, processes and how-to guides
- Sales enablement, answering product questions from specifications and case studies
- Document search across contracts, reports or research
- Onboarding, helping new staff find answers without interrupting colleagues
RAG vs fine-tuning vs a general assistant
| Approach | Best for | Updates | Shows sources |
|---|---|---|---|
| General AI assistant | General questions and writing | N/A | Sometimes |
| RAG | Answering from your own, changing information | Easy: update the documents | Yes |
| Fine-tuning | Teaching a model a style, format or specialised task | Requires retraining | No |
Many systems combine approaches, but RAG is usually the right starting point for business knowledge.
What makes RAG accurate
- Good source content: current, clear, non-contradictory documents
- Sensible chunking: passages large enough to keep context, small enough to stay precise
- Strong retrieval: combining meaning-based and keyword search often helps
- Clear instructions: answer only from the sources, and say "I don't know" when unsure
- Citations: link every answer to its source
- Evaluation: test with a set of real questions and expected answers
- Access control: people only get answers from documents they're allowed to see
Limitations to understand
- If the documents are wrong or outdated, answers will be too
- Questions needing calculation or reasoning across many documents can be harder
- Poor retrieval leads to confident answers based on the wrong passage
- Sensitive documents need careful permission handling
- Ongoing maintenance is needed as content changes
Questions to ask before building
- What questions should it answer, and for whom?
- Where does the knowledge live today, and who keeps it current?
- Who should be able to see which information?
- What should happen when it doesn't know?
- How will we measure accuracy?
Costs in practice
Most of the effort is in preparing content, getting retrieval right and testing, rather than in the AI model itself. Running costs depend on the volume of questions and documents. A focused pilot, on one team or one set of documents, is the sensible way to start.
Example: an HR policy assistant
A 60-person company wants staff to get quick answers to HR questions.
| Design decision | Choice |
|---|---|
| Sources | Employee handbook, holiday policy, benefits guide, expenses policy |
| Access | All staff can see general policies; managers also see management guides |
| Where it lives | Inside the company's chat tool |
| Answers | Short, with a link to the exact policy section |
| When unsure | Says so and suggests contacting HR |
| Updates | HR updates documents; the index refreshes automatically |
| Measurement | Accuracy checks on 50 common questions, plus a thumbs up or down on each answer |
The result: staff get instant, sourced answers, and HR spends less time on repeat questions.
How to evaluate a RAG system
| What to test | How |
|---|---|
| Retrieval accuracy | Does it find the right passage for each test question? |
| Answer accuracy | Is the answer correct and complete? |
| Citations | Do sources actually support the answer? |
| Refusals | Does it say "I don't know" when the answer isn't in the documents? |
| Permissions | Does it respect who can see what? |
| Speed | Are answers fast enough for everyday use? |
Build a test set of real questions with agreed answers, and rerun it whenever documents or settings change.
How we help
Our internal knowledge assistants and AI chatbots and assistants are built on RAG, with citations, access controls and evaluation from day one. For broader context, read our guides to AI agents and AI chatbots for small business.
The short version
RAG lets AI answer from your own documents by retrieving relevant passages before generating an answer, ideally with citations. It's easier to update than fine-tuning and suits support, internal knowledge and sales. Accuracy depends on good content, retrieval, clear instructions, testing and access controls.
