The Impact of Large Language Models vs. Web Searches on Learning

by Tim Leogrande, BSIT, MSCP, Ed.S.

9 JAN 2026 • 3 MIN READ

Since ChatGPT was made available to the public in 2022, millions of people have begun using large language models (LLMs) to get work done and access information. The appeal is obvious; ask a question, get a polished summary, and move on.

But a recent study by Jin Ho Yun of New Mexico State University and Shiri Melumad of the University of Pennsylvania reports that this efficiency may come at a price. Their findings indicate that students who use LLM-generated summaries tend to develop a shallower understanding of the subject matter than those who use regular Google searches. The authors' findings are derived from an analysis of seven research studies involving over 10,000 participants, and the methodological framework was comparable across most of these papers.

Participants were told to learn about a certain topic and were then randomly assigned to do so by using either an LLM like ChatGPT, or by using the links returned by a standard Google web search. They were not limited in their utilization of these tools. Those using Google could perform as many searches as they wanted, and those using an LLM could ask as many questions and follow-up queries as they deemed necessary. After they completed their research, participants were asked to write a piece of advice for a friend about what they had learned.

<aside> 💡

The data indicated a clear pattern. Participants who learned through an LLM had a more shallow understanding of the material than those who used a standard web search.

</aside>

The LLM group had put in less effort, which led to advice that was briefer, less accurate, and more general. When their advice was given to a group of readers who didn't know which tool had been used, they found that the LLM-based advice was less useful and not as informative; and that the readers were less likely to act upon it. One possible reason why LLM-generated advice is shallower and more generic is that LLM outputs provide users less heterogenous information than the diverse sources that are usually returned by a Google search.

Why did the LLM group exhibit shallower learning? One of the cornerstones of education is that people learn better when they are actively engaged with the material. Traditional web searches usually introduce more friction into the learning process because students have to follow several links, look at primary sources, and figure out how to synthesize information from many different web pages on their own. This friction makes learning more labor intensive, but it also helps students build more complex, specific, and original mental models of the material. LLMs, on the other hand, do a lot of this heavy lifting for the user, making the learning process more passive.

The researchers don't assert that LLMs should be avoided completely, which is important because these models offer a lot of benefits across several domains. Instead, they suggest that people should be more careful and deliberate about how they utilize these technologies. This starts with figuring out what kinds of learning goals and domains of knowledge LLMs are likely to be most helpful with.

It may be perfectly fine to use an LLM co-pilot for brief, factual questions. But if the goal is to develop deep, flexible, and generalizable understanding, relying on LLM-generated summaries isn’t as effective.

As part of their larger research program, the authors are exploring whether LLM-based learning may be made more active. In one experiment, they looked at a customized GPT model that provided real-time web links along with the LLM’s synthesized answers.

<aside> 💡

Even in this situation, participants displayed little interest in looking at original sources after they had read a synthesized LLM summary. As a result, these individuals gained shallower knowledge than those who used a Google search.

</aside>

Next, the researchers intend to examine LLM models that introduce some helpful friction into the learning process. In particular, they want to find out which hurdles or guardrails work best to incentivize users to go beyond simple summaries and engage more deeply with the subject matter.

These tools may be especially useful in high schools, where teachers face the Sisyphean task of helping students learn the basics of reading, writing, and math while also preparing them for a world where LLMs are likely to be ubiquitous.