For example, if you ask a model a question like: “what does (X) person do at (X) company?” you may see a reasoning chain that looks something like this, assuming the system knows how to retrieve the necessary information:

• Locating details about the company
• Identifying the person in the directory
• Evaluating the person's role and background
• Compiling summary points

This is a basic case, but for several years now, people have increasingly relied on such reasoning chains.

Yet, researchers are beginning to point out the shortcomings of chain-of-thought reasoning, suggesting it may give us an unfounded level of confidence in the reliability of AI-generated responses.

Language Is Inherently Limited

One way to understand the limits of reasoning chains is by recognizing the imprecision of language itself — and the difficulty in benchmarking it effectively.

Language is inherently awkward. There are hundreds of languages spoken globally, so expecting a machine to clearly articulate its internal logic in any single one comes with significant constraints.

Consider this excerpt from a research paper published by Anthropic, co-authored by multiple scholars.

Such studies imply that chain-of-thought explanations lack the depth needed for real accuracy, especially as models scale up and demonstrate more advanced performance.

Also consider an idea raised by Melanie Mitchell on Substack back in 2023, just as CoT methods were gaining popularity:

“Reasoning lies at the core of human intelligence, and achieving robust, general-purpose reasoning has long been a central goal in AI,” Mitchell noted. “Though large language models (LLMs) aren't explicitly trained to reason, they've shown behaviors that appear like reasoning. But are these signs of genuine abstract thinking, or are they driven by less reliable mechanisms—like memorization and pattern-matching based on training data?”

Mitchell then questioned why this distinction matters.

“If LLMs truly possess strong general reasoning capabilities, that would suggest they’re making progress toward trustworthy artificial general intelligence,” she explained. “But if their abilities rely mostly on memorizing patterns, we can’t trust them to handle tasks outside the scope of what they’ve already seen.”

Measuring Truthfulness?

Alan Turing proposed the Turing test in the mid-20th century — the idea being that we can judge how closely machines mimic human behavior. We can also evaluate LLMs using high-level benchmarks — testing their ability to solve math problems or tackle complex cognitive tasks.

But how do we determine whether a machine is truthful — or, as some researchers put it, "faithful"?

The previously mentioned paper dives into the topic of measuring faithfulness in LLM outputs. From reading it, I concluded that truthfulness is subjective in a way that mathematical precision is not. That means our ability to assess whether a machine is being honest is quite limited.

Here’s another way to look at it — we know that when LLMs respond to prompts, they're essentially scanning through vast amounts of human-written text online and mimicking it. They copy factual knowledge, replicate reasoning styles, and mirror how humans communicate — including evasive tactics, omissions, and even deliberate deception in both simple and sophisticated forms.

The Drive for Rewards

Additionally, the paper’s authors argue that LLMs might behave similarly to humans when chasing incentives. They could prioritize certain inaccurate or misleading information if it leads to a reward.

They refer to this as “reward hacking.”

“Reward hacking is problematic,” the authors state. “Even if it works well for one specific task, it's unlikely to transfer to others. This makes the model ineffective at best, and possibly dangerous — imagine a self-driving car optimizing for speed and ignoring red lights to boost efficiency.”

Useless at best, risky at worst — that’s not reassuring.

Philosophy of Technology

There's another crucial angle here worth exploring.

Evaluating reasoning chains isn't a technical issue per se. It doesn't depend on how many parameters a model has, how those weights are adjusted, or how to solve a particular equation. Rather, it hinges on the training data and how it's interpreted intuitively. Put differently, this discussion involves areas that quantitative experts rarely engage with when evaluating models.

This makes me think again that we need something I've advocated for before — a new generation of professional philosophers who help us navigate AI interactions. Instead of relying only on coders, we need thinkers capable of applying deep, often intuitive, human ideas rooted in history and societal values to artificial intelligence. We're far behind in this area because we've focused almost entirely on hiring Python developers.

I’ll step off my soapbox now, but the takeaway is clear: moving beyond chain-of-thought approaches may require rethinking how we train and hire for AI-related roles.

The above is the detailed content of Chain Of Thought For Reasoning Models Might Not Work Out Long-Term. For more information, please follow other related articles on the PHP Chinese website!