The problem RAG solves
A language model was trained on public text, up to a given date. It knows neither your contracts, nor your procedures, nor your client history. Ask it a question about your company and it will produce a plausible, wrong answer, with the same confidence as a correct one. That is the behaviour that durably burned AI in the eyes of many business leaders.
RAG, for Retrieval-Augmented Generation, fixes this with a simple idea: you do not ask the model to know, you give it something to read. Before answering, the system searches your documents for the relevant passages, then asks the model to answer based solely on those passages, citing its sources.
How it works, without jargon
- Your documents are split into reasonably sized chunks, a paragraph or a section. A 40-page contract becomes a few dozen self-contained pieces.
- A meaning fingerprint is computed for each chunk. It is a series of numbers that places the text in a space where two texts with similar meaning sit close together. That is what lets the system find "termination notice period" when the question says "how do we stop the contract".
- Those fingerprints are stored in a specialised database.
- When a question comes in, the same fingerprint is computed and the closest chunks are retrieved.
- The answer is assembled: the model receives the question and those chunks, with the instruction to answer from them and to say so when the information is not there.
The most important practical consequence: a well-built RAG system knows how to answer "this information is not in the documents". That property is what makes it usable in a company, and it is the first thing to test in a demo. Ask a question whose answer you know exists nowhere in your files: if the system makes something up, it is not ready.
What it changes concretely in an SME
- Internal search becomes a conversation. Instead of hunting for the right file, you ask the question and get the answer with a link to the source.
- Onboarding speeds up. A newcomer queries the company's memory instead of pulling a colleague away for every question.
- Expert knowledge survives departures, provided it was written down somewhere first.
- Client answers standardise on what is actually written in your contracts and procedures, not on each person's recollection.
What it takes to succeed, and it is demanding
RAG does not compensate for missing material. It is the point we repeat most often during scoping, because it determines the outcome more than any technical choice.
- You need documents. If the knowledge lives in people's heads, it must be written down first. No tool reads minds.
- You need a single version of the truth. Three versions of a procedure in three different folders produce a system that answers at random, and a random answer is worse than no answer.
- You need access rights handled. An assistant that answers everyone about salaries is an incident, not a feature. Rights must apply to the search, not just the display.
- You need maintenance. An updated document must be re-indexed, otherwise the system answers with the previous version, without flagging it.
Where your data goes
Three things potentially leave your walls: the documents sent for fingerprint computation, the user's question, and the passages passed to the model to write the answer. Depending on your requirements, everything can stay in-house with self-hosted models, or only the generation step can be externalised to an API whose contractual terms exclude training on your data. Those are two different levels of cost and quality, and it is a management decision, not a technical one.
Our default practice: the document base and the index stay on your infrastructure or on servers in France, and generation goes through an API with a contractual commitment of non-reuse. When sensitivity demands it, everything stays internal.
Should you do RAG, or something else?
RAG is good at questions whose answer is written somewhere. It is bad at three families of needs, and confusing them is the most frequent cause of failure.
- Computation questions on structured data ("what is this month's revenue") belong to a query on your database, not to semantic search.
- Questions that require synthesising the whole corpus ("what are the recurring themes in our complaints") call for batch processing, not a search for the ten closest passages.
- Actions ("create the quote") belong to an automation with rules, possibly triggered by the assistant, but not to RAG itself.
In practice, a useful system combines all three: document search for what is written, queries for what is numeric, and automation for what must be done. The point is not to pick a technique, it is to know which one applies to which question.
Where to start
Choose a narrow, high-value corpus: one department's procedures, one product's documentation, the history of client answers. Check that the corpus is up to date and unique. Build a first assistant on that scope, measure the rate of correct answers on thirty or so real questions asked by the teams, and only then expand. An assistant that is excellent on one domain gets adopted; an assistant that is mediocre on everything is abandoned within three weeks.
Frequently asked questions
Does RAG completely prevent AI errors?
No, it strongly reduces them and makes them verifiable. The model can still misread a passage, and the search can miss the right document. The decisive difference is that every answer cites its sources: a user can check in one click, which is impossible with a model answering from memory.
How many documents does it take to be worth it?
There is no threshold in number of files. The criterion is the number of recurring questions those documents answer. A hundred pages of procedures that people actually consult justify an assistant; ten thousand pages nobody reads do not.
Is our data used to train the model?
Not if the architecture is correct. In a RAG system, your documents are passed to the model at question time, as reading context, and are not folded into its training. What remains is to check the API provider's contractual terms on the reuse of transmitted data, which can be verified and negotiated.
How long does it take to get a first assistant into service?
On a narrow corpus that is already up to date and centralised, count on the AI Sprint format: two to five weeks between scoping and production. If the documents first need to be gathered, deduplicated and updated, that preliminary work often lasts longer than the technical build.
Wondering whether your document base holds up?
That is exactly what the diagnostic looks at: the real state of your documents and the corpus to favour for a first assistant. Twenty minutes to talk it through.
Book a free diagnostic