Retrieval-augmented generation (RAG)

NeurIPS 2020: Patrick Lewis and coauthors at Facebook AI presented the model that gave the technique its name, a generator paired with a dense vector index of Wikipedia that a neural retriever queries before each answer. Retrieval-augmented generation is an architecture that combines a large language model with a document-retrieval mechanism. Rather than answering solely from the knowledge frozen in its parameters, the model first queries a base of relevant documents and then grounds its answer in the retrieved extracts. The benefit is twofold, grounding answers in verifiable, up-to-date sources, which appreciably reduces the risk of hallucination, and allowing the system to reason over private or recent information the model never saw during training. This technique has become one of the most practical ways to make AI-based enterprise applications reliable. It does, however, introduce a new attack surface, because the retrieved documents become a route of entry for malicious instructions. An attacker who manages to insert booby-trapped content into the queried base can hijack the system's behavior through indirect prompt injection, the model treating as an instruction what was supposed to be mere data. Retrieval-augmented generation thus illustrates the permanent trade-off of AI security, in which every gain in capability opens a new vulnerability.