RAG Isn't Dead, It Just Grew Up

In 2023 we talked about RAG. In 2026 we talk about agents. But in any serious agent, the same old problem when working with information is still alive: what to retrieve, when, and at what cost.

This article reviews the scientific literature to test two hypotheses that argue for RAG's obsolescence: the redundancy of retrieval given extended context windows (1M+ tokens), and the displacement of this technique by agent-based systems. The data says otherwise on both counts, and it also explains why ignoring them costs you dearly.

If your system still retrieves information the way it did in 2023, this is for you. The question is no longer whether RAG is dead. It's how much context budget you're wasting without realizing it.

Has RAG really died?

You've probably seen the same headline on LinkedIn, "X," or in some technical newsletters: "RAG is dead." It's said with the same conviction as in 2022, when we were promised that prompt engineering would be the profession of the future, or in 2025, when we were assured that autonomous agents would leave us all jobless. Like every tech prophecy, there's some truth to it, but also a healthy dose of posturing.

Let's do what we do best: debunk the myths with data, not opinions.

In 2023, RAG (Retrieval-Augmented Generation) was a simple architecture: a question, an embedding, a fragment search (with luck, a re-ranker), a prompt, and done. It was a series of boxes with arrows, one after another, that got companies excited just by seeing it on a slide. In 2026 that architecture is history. Models handle context windows of a million tokens or more, agents grep through repositories, run SQL, call APIs, reformulate a search if the first one doesn't work, and carry working memory across steps. The four-box diagram no longer describes almost any real system.

Hence the controversy of the headline. But my position is different: RAG has not died. It has stopped being a closed-off solution to become a basic piece of engineering. The fact that it's just one more tool in the system, rather than the whole system, is proof that the technology has matured.

From an objective standpoint, what part of RAG really has died

Before defending RAG, I want to be honest with those who consider it buried, since they have a point. There are pieces of the 2023-era RAG that make no sense in a serious system today. I'm thinking, for example, of fixed top-k with no justification, of 512-token chunks just because, of using a single embedding as a universal search engine, or of relying on dense retrieval with no lexical component. That linear pipeline of five boxes in a row is, indeed, what has died.

There is a compelling technical reason for this shift. More than a failure of power, we've become aware of limits we used to ignore. When generating embeddings, we face an almost impossible balancing act: capturing a text's overall meaning while also distinguishing very fine-grained nuances. This is the so-called "granularity dilemma." Text encoders systematically fail when two passages share the same semantic field but differ in one specific entity or fact.

The interesting part is that model size doesn't solve this on its own. A recent study shows that a model of just 0.1B parameters, fine-tuned specifically for this task, outperforms 7B models (Xu et al., 2025). The failure isn't one of power, it's one of design. That's why a single vector search channel never was, and still isn't today, a solid foundation.

The exact same thing happens with chunking. Splitting into large blocks mixes different entities into the same vector and dilutes the nuance; splitting into tiny fragments gains precision on the entity but loses the context that gives it meaning. That's why the solutions that work in production don't look for an "optimal chunk size," but instead use schemes like hierarchical chunking: index the small fragment, but retrieve the paragraph or document that surrounds it.

Long context has also genuinely improved, and I won't dispute that with anyone. There are models today that support windows of one, two, or even ten million tokens, and this obviously shifts the rules a bit as to what's worth retrieving as a unit. Before, you had to chop up a document because it didn't fit whole. Now, in many cases, you can fit the whole document in. That's not marketing, it's a real architectural improvement. But this doesn't justify dumping the entire corpus into a prompt, and this is where the "infinite context" narrative starts to collide with the numbers and with reality.

Myth 1: "Giant context windows make RAG unnecessary"

This is the argument that grabs the most headlines: if models already handle windows of one or two million tokens, why bother retrieving anything? Just inject everything. The technical problem is that a model doesn't pay attention uniformly across that entire window. This phenomenon is known as Context Rot (or context degradation): the more information we inject, the more noise we introduce, and the worse the model reasons (Liu et al., 2024).

The RULER benchmark exposed this mirage back in 2024. At that point, although most models claimed to support windows of 32K tokens or more, only 50% maintained acceptable performance under real-world usage conditions (Hsieh et al., 2024).

The result that, in my view, most changes the perspective on this problem is NoLiMa. While RULER usually lets the model find the answer through mere lexical matching, NoLiMa removes that "hint" and forces the model to reason semantically. The results were revealing: of 13 models claiming to support contexts of at least 128K tokens, 11 dropped below 50% of their performance by the time they reached 32K. Even a benchmark model like GPT-4o went from 99.3% accuracy in short context to 69.7% under long-context conditions (Modarressi et al., 2025)*.

*Given the dizzying pace of new model development, it's essential to verify the specific versions and architectures referenced in Modarressi et al., 2025, before generalizing their results.

We should also consider that almost all of this evidence is measured in English. When this capability is tested across 26 languages at once, the performance gap between high-resource and low-resource languages triples as the window grows (Kim et al., 2025; Hengle et al., 2026; Wang et al., 2026; Qi et al., 2025).

All of this evidence points in the same direction:

Just because a model can afford the cost of reading a million tokens doesn't mean that million tokens is useful attention.

That's why I prefer to see retrieval not as a technique competing with long context, but as the manager of our budget. Every call to the model is a finite resource that we must divide between instructions, history, memory, and retrieved documents. Retrieval is the function that decides which part of the world deserves to occupy that space. The bigger the window, the more expensive it is to waste it. Retrieving information precisely is more critical today than ever.

Myth 2: "Agents replace RAG"

This is, in my view, the strongest argument from those who consider RAG dead. Let's examine it closely, though. Most agents decide their next step based on what they just observed, not on a fixed script, and at some point in that loop, the same question almost always comes up: what do I need to know before continuing? That question is retrieval, even if it no longer carries that label.

An agent doesn't follow a linear flow. It must plan, search, evaluate results, decide whether it needs more information, consult tools, and compare before writing a single word. Even when the search strategy becomes multi-level, the operational essence remains data retrieval. What has evolved isn't the act of retrieving itself, but the system's orchestrating intelligence to discern which channels to consult and exactly when to execute each strategy.

Current evidence shows that letting an agent perform exhaustive searches isn't just costly, it's often counterproductive. A recent benchmark, ContextBench, analyzed the behavior of different models (GPT-5, Claude Sonnet 4.5, and Devstral 2) operating as autonomous agents on hundreds of real incidents. The results dismantle the intuition that "more searching equals a better answer."

The analysis is clear: aggressive exploration, where the agent runs excessive rounds of search, drives up token consumption and operational cost, and, moreover, doesn't guarantee higher quality (Li et al., 2026). By contrast, models that adopt a moderate approach achieve a superior balance and reach better performance without needing to saturate the system with redundant queries. The lesson from all this is that aggressive exploration by agents only inflates spend, not accuracy. Reading more doesn't equal retrieving better.

In practice, what we're seeing consolidate in 2026 are hybrid pipelines that optimize resources. A successful pattern usually looks like this:

Initial retrieval: combine sparse representations (BM25) with dense embeddings to get a broad set (50-100 candidates).
Re-ranking: pass that set through a cross-encoder that computes real cross-attention between the query and each fragment.
Selection: keep only the 5 to 10 most relevant blocks before calling the generator model.

This approach is more expensive than a simple embedding, yes, but infinitely more efficient than saturating the LLM with a hundred random fragments. As confirmed by the study by Assadi et al. (2026), in semantic search, a good embedding model beats any combination that tries to use an LLM to re-rank results from scratch. Spending the expensive model where it isn't needed not only increases cost; it often doesn't even improve the result.

Another successful pattern, based on an architecture built around a reflective reasoning loop, usually looks like this:

Decomposition and planning: the agent receives the complex query and unfolds a dynamic planning graph to break the problem into sub-tasks, avoiding static sequences.
Intent routing: it evaluates the type of data required and routes each sub-task to the appropriate specialized retrieval tools (such as vector engines, structured SQL queries, or knowledge graphs).
Execution and retrieval: it runs queries against the selected channels to extract the necessary empirical evidence.
Self-reflection: it recursively assesses whether the retrieved evidence is sufficient, accurate, and relevant to resolve the query.
Automatic re-planning: if the internal relevance or data-sufficiency metric fails, the system triggers automatic re-planning to iterate or search new sources.
Evidential reasoning: it consolidates and adjusts its strategy in real time, using validated evidence to synthesize the final answer.

Obviously, this orchestrating sophistication raises a first-order pragmatic challenge: accumulated latency and cost per query. However, delegating every micro-decision of this loop to a frontier LLM is economically unsustainable. Architectures that scale apply a layered approach: they use embedding models and re-rankers (cross-encoders) to filter context efficiently, reserving the heavy LLM only for the final synthesis phase. This specialized filtering drastically reduces noise and operational cost, ensuring that only the highest-quality evidence reaches the reasoning model.

On the other hand, when a query requires connecting related concepts across multiple documents, vectors fail. This is where knowledge graphs solve the equation, allowing the model to reason over explicit connections.

These have proven superior to pure vector search (Pan et al., 2024; Chen et al., 2024). In the past, the problem was the cost of building the entire graph. But recent implementations like GraphRAG (Edge et al., 2024) and optimized solutions such as LazyGraphRAG (Edge et al., 2024), HippoRAG (Gutiérrez et al., 2024), or LightRAG (Guo et al., 2025) allow complex semantic relationships to be extracted with reduced indexing costs, with GraphRAG-bench (Xiao et al., 2025) serving as a key reference for evaluating reasoning capability in these environments.

Ultimately, this ecosystem of techniques (from hybrid retrieval to orchestration via agents and knowledge graphs) doesn't offer universal solutions. Success lies in the modular composition of these mechanisms according to the complexity and intent of each query.

So the useful debate in 2026 is no longer "RAG or agents?" The right question is: which retrieval mechanism should each agent use, at what point in its reasoning, and with what budget? The agent doesn't replace RAG, it absorbs it as an internal capability.

The metric we tend to forget: latency and cost

In corporate environments, the bottleneck usually isn't the technology, it's the budget. Forcing models to process immense contexts by default wrecks the economic viability of any product. Precision in filtering information isn't just a technical challenge, it's financial discipline so that the cloud bill doesn't eat into the business margin.

A million tokens costs money and time. Meanwhile, an efficient hybrid search filters data in milliseconds. Forcing the model to process the entire repository before writing the first word drives latency up to several seconds. For a conversational assistant, sending hundreds of thousands of tokens per call isn't just expensive, it's a terrible user experience. The goal in production is never to saturate the model with superfluous data, but to deliver, with great precision, the few tokens that actually determine the decision.

The providers themselves confirm this by launching features like context caching (Gemini's or Claude's context cache). These tools are a complement to smart retrieval, never a substitute, designed precisely so you don't have to pay for or wait on the same data over and over.

The data backs up this caution. Assadi et al. (2026) show that, although LLMs achieve competitive results, the cost is disproportionate. Using a model like Gemini 3.1 Pro for retrieval tasks can be up to 1,431 times more expensive than using a specialized embedding model. Moreover, LLM processing speed is drastically lower. In terms of efficiency, tokens devoted to "reasoning" account for between 28% and 81% of the total cost. Often, the model goes back and forth over relevance judgments it had already settled on the first read, inflating the bill without improving the result.

The lesson is clear:

Use embedding models for the bulk of the work and reserve the LLM for the final reasoning that truly needs it.

The question for anyone designing a generative AI system should no longer be "how many tokens does my model support?" but "what is the optimal context budget for each step of my system?"

So, what do we do with our 2023 RAG?

It's time to set aside theoretical debates and be pragmatic. If your current system is still anchored in 2023's assumptions, such as static chunking, blind vector search, and a rigid linear flow, it's only natural that the results fall short.

The problem isn't that RAG has died. The problem is that you're trying to solve today's challenges with an obsolete architecture. Your pipeline needs to stop being a simple search engine and become a decision-making system.

To modernize it, any production solution must answer these six critical questions:

What to retrieve? Precisely isolate the exact information that resolves the query, going beyond simple similarity.
When to act? Determine the precise moment for the query so as not to saturate the system with unnecessary searches.
How to search? Choose the right path (vector, lexical, structured, temporal, hybrid, or agent-assisted) based on the nature of the query.
Who has access? Guarantee security filters and user permissions before injecting any data into the context.
Is it still current? Validate the temporal freshness of the information to avoid feeding the model outdated data.
Where does it come from? Ensure full traceability and exact attribution of sources, a non-negotiable requirement for business decisions.

My hypothesis for 2026/2027

The term "RAG" has lost the spotlight to the agent narrative, but information retrieval hasn't disappeared. It has simply stopped being a standalone component. It no longer needs a name of its own because it is, quite simply, an inherent capability.

Retrieval is the connective tissue of the agent. Whether querying a database, a document, code, or an internal policy, the flow is constant: the agent determines the need, the system validates access, the retriever filters the content, and the model reasons over the selected information.

That's why "has RAG died?" strikes me as the wrong question for 2026. The real unknown we need to solve in every project is more technical and less about appearances: how much context budget are we wasting?

RAG should no longer be understood as a rigid architecture drawn on a slide, it's a basic function of any generative AI system. The bigger the context windows get, the more vital it becomes to execute this function with precision.

So:

What has died isn't RAG. It's your 2023 pipeline.

If you found this review useful, share it with anyone still arguing over whether "RAG is dead" instead of discussing how their system should be retrieving.

References

Assadi, A. E., Muennighoff, N., & Lee, J. (2026). The Embedder's Dilemma: LLMs Are Better, but at What Cost?. arXiv preprint arXiv:2608.12875.
Chen, Z., Zhang, Y., Fang, Y., Geng, Y., Guo, L., Chen, X., ... & Chen, H. (2024). Knowledge graphs meet multi-modal learning: A comprehensive survey. arXiv preprint arXiv:2402.05391.
Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., ... & Larson, J. (2024). From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130.
Edge, J. L. D., Trinh, H., & Larson, J. (2024). Lazygraphrag: Setting a new standard for quality and cost. Microsoft Blog.
Guo, Z., Xia, L., Yu, Y., Ao, T., & Huang, C. (2025, November). LightRAG: Simple and Fast Retrieval-Augmented Generation. In EMNLP (Findings) (pp. 10746-10761).
Gutiérrez, B. J., Shu, Y., Gu, Y., Yasunaga, M., & Su, Y. (2024). Hipporag: Neurobiologically inspired long-term memory for large language models. Advances in neural information processing systems, 37, 59532-59569.
Hengle, A., Bajpai, P., Dan, S., & Chakraborty, T. (2026, March). Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval over haystacks. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 6128-6152).
Hsieh, C. P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., ... & Ginsburg, B. (2024). RULER: What's the real context size of your long-context language models?. arXiv preprint arXiv:2404.06654.
Kim, Y., Russell, J., Karpinska, M., & Iyyer, M. (2025). One ruler to measure them all: Benchmarking multilingual long-context language models. arXiv preprint arXiv:2503.01996.
Li, H., Zhu, L., Zhang, B., Feng, R., Wang, J., Pan, Y., ... & Ye, H. (2026). Contextbench: A benchmark for context retrieval in coding agents. arXiv preprint arXiv:2602.05892.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12, 157-173.
Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schütze, H. (2025). Nolima: Long-context evaluation beyond literal matching. arXiv preprint arXiv:2502.05167.
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024). Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, 36(7), 3580-3599.
Qi, J., Fernández, R., & Bisazza, A. (2025, November). On the consistency of multilingual context utilization in retrieval-augmented generation. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025) (pp. 199-225)
Su, H., Yen, H., Xia, M., Shi, W., Muennighoff, N., Wang, H. Y., ... & Yu, T. (2025, May). Bright: A realistic and challenging benchmark for reasoning-intensive retrieval. In International Conference on Learning Representations (Vol. 2025, pp. 48941-48991).
Wang, D., Mo, G., Shi, Y., Zhang, C., Zheng, B., Cao, B., ... & Sun, L. (2026, July). All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAG. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 7441-7455).
Xiao, Y., Dong, J., Zhou, C., Dong, S., Zhang, Q. W., Yin, D., ... & Huang, X. (2025). Graphrag-bench: Challenging domain-specific reasoning for evaluating graph retrieval-augmented generation. arXiv preprint arXiv:2506.02404.
Xu, L., Su, Z., Yu, M., Li, J., Meng, F., & Zhou, J. (2025). Dense retrievers can fail on simple queries: Revealing the granularity dilemma of embeddings. Passages, 3024, 8.