The RAG Plateau: Where Retrieval-Augmented Generation Actually Pays, and Where It Was Oversold
Two years into the retrieval-augmented generation build cycle, the pattern of where RAG earns its keep and where it has been quietly retired is now visible.
Retrieval-augmented generation was the default enterprise AI architecture for most of 2023 and 2024. Ground the model in your own documents, the argument went, and you get accuracy and provenance without the cost of fine-tuning. Two years and several thousand production deployments later, a more honest map has emerged. RAG works, but only for a narrower set of use cases than the initial wave assumed, and the enterprise architects we speak to have started to sort their portfolios into three buckets: keep and invest, keep as-is, and retire.
The keep-and-invest bucket is dominated by internal knowledge retrieval against corpora that are large, updated frequently, and consumed by employees with domain expertise who can spot a wrong answer. Legal document search inside law firms, engineering document search inside industrial companies, and internal policy lookup inside large enterprises all sit here. In these settings, the retrieval layer is doing genuine work that the model alone cannot do, the users are equipped to verify outputs, and the marginal cost of a bad answer is low because a human is in the loop. The teams that have kept investing in RAG for these use cases have moved past naive vector search to hybrid retrieval that combines dense embeddings with structured metadata, and they are getting real productivity gains that show up in time-to-answer for expensive knowledge workers.
The keep-as-is bucket is customer-facing FAQ and support triage against a well-curated knowledge base. Here RAG works, but the ceiling is lower than the initial pitches suggested. The retrieval layer helps, but the real determinants of quality are the curation of the knowledge base and the design of the fallback path when the model does not find a confident answer. The organizations that have gotten good results in this bucket have generally invested more in the knowledge-base curation and the escalation design than in the retrieval architecture itself. That is a management fact more than a technology fact, and it is often the wrong team that owns it.
The retire bucket is where the honest reckoning has happened. Public-facing chat interfaces built on RAG against loosely curated web content, generative research assistants that were expected to synthesize across contradictory sources without human judgment, and complex analytical tasks where the answer requires reasoning across many retrieved chunks: these have quietly been walked back or replaced with either longer-context models that do not need external retrieval, or with more structured pipelines that pre-process the retrieval into a form the model can actually reason about. The RAG-as-magic assumption did not survive contact with users who noticed the answers were wrong in ways that undermined trust in the whole product.
The frontier-model context-window expansion is the other structural change. When models can hold hundreds of thousands of tokens of context reliably, a class of use cases that previously required retrieval can now be handled by loading the relevant material directly into the prompt. This is not free, and the cost-per-query math still favors retrieval for large corpora, but for medium-sized document sets, the operational simplicity of just loading the document is winning. Architecture teams that built elaborate retrieval pipelines for what turned out to be a fifty-document problem are now removing complexity, and the ones being honest about it are reporting better outputs, not worse.
For a Chief Architect or head of AI engineering, the practical guidance is to run the portfolio against three questions. First, is the corpus large, dynamic, and metadata-rich enough that retrieval is doing real work? If not, longer-context prompting is probably the simpler answer. Second, are the users equipped to catch wrong answers, and is the fallback path designed for the case where the model is not confident? If not, RAG is likely producing a false sense of grounding rather than actual grounding. Third, is the knowledge-base curation owned by someone with authority to say no to bad content? If not, no retrieval architecture will save the output quality.
The broader observation is that RAG was not oversold as a technology. It was oversold as a substitute for the harder work of curating the underlying knowledge and designing the human workflow around it. The teams that treated retrieval as an engineering shortcut are the ones now removing it. The teams that treated it as one component of a larger information-architecture problem are the ones still investing.