Jing Hu isn’t your typical AI commentator. Trained as a biochemist, she traded the lab bench for the wild west of tech, spending a decade building products before turning her sights on AI research and journalism. Hu’s publication on Substack, “2nd Order Thinkers,” examines AI’s impact on individuals and the commercial world, as Hu states, “thinking for yourself amid the AI noise.” In a recent episode of “Tech Uncensored” I spoke with Hu to discuss the cognitive impacts from the increased use of chatbots built on large language models.

Chatbots like Gemini, Claude, and ChatGPT continue to herald significant progress but are still wrought with inaccurate, nonsensical and misleading information — hallucinations. The content generated can be harmful, unsafe, and often misused. LLMs today are not fully trustworthy, by the standards we should expect for full adoption of any software products.

Are Writing and Coding Occupations at Risk?

In her recent blog, “Why thinking Hurts After Using AI,” Hu writes, “Seduced by AI’s convenience, I’d rush through tasks, sending unchecked emails and publishing unvetted content,” and surmises that “frequent AI usage is actively reshaping our critical thinking patterns.” Hu references OpenAI and UPenn study from 2023 that looks at the labor market impact from these LLMs. It states that tasks that involve science and critical thinking are the tasks that would be safe; however, those which involve programming and writing would be at risk. Hu cautions, “however, this study is two years old, and at the pace of AI, it needs updating.”

She explains, “AI is very good at drafting articles, summarizing and formatting. However, we humans are irreplaceable when it comes to strategizing or discussing topics that are highly domain specific. Various research found that AI’s knowledge is only surface level. This becomes especially apparent when it comes to originality.” Hu explains that when crafting marketing copy, “we initially thought AI could handle all the writing. However, we noticed that AI tends to use repetitive phrases and predictable patterns, often constructing sentences like, 'It’s not about X, it’s about Y,' or overusing em-dashes. These patterns are easy to spot and can make the writing feel dull and uninspired.”

For companies like Duolingo whose CEO promises to be an “AI-first company,” replacing their contract employees is perhaps a knee-jerk decision that has yet to be brought to bear. The employee memo clarified that “headcount will only be given if a team cannot automate more of their work.” The company was willing to take “small hits on quality than move slowly and miss the moment.”

For companies like this, Hu argues that they will run into trouble very soon and begin rehiring just to fix AI generated bugs or security issues. Generative AI for coding can be inaccurate because models were trained on Github, or similar databases. She explains, “Every database has its own quirks and query syntax, and many contain hidden data or schema errors. If you rely on AI-generated sample code to wire them into your system, you risk importing references to tables or drivers that don’t exist, using unsafe or deprecated connection methods, and overlooking vital error-handling or transaction logic. These mismatches can cause subtle bugs, security gaps, and performance problems—making integration far more error-prone than it first appears.”

Another important consideration is cybersecurity, which must be approached holistically. “If you focus on securing just one area, you might fix a vulnerability but miss the big picture,” she said.

She points to the third issue: Junior developers using tools like Copilot often become overly confident in the code these tools generate. And when asked to explain their code, many are unable to do it because they don’t truly understand what was produced.

Hu concedes that AI is good at producing code quickly, however it is a only part (25% to 75%) of software development, “People often ignore the parts that we do need: architecture, design, security. Humans are needed to configure the system properly for the system to run as a whole.” She explains that the parts of code that will be replaced by AI will be routine and repetitive, so this is an opportune moment for developers to transition, advising “To thrive in the long term, how should we — as thinking beings —develop our capacity for complex, non-routine problem-solving? Specifically, how do we cultivate skills for ambiguous challenges that require analysis beyond pattern recognition (where AI excels)?”

The Contradiction of Legacy Education and The Competition for Knowledge Creation

In a recent New York Times article. “Everyone is Cheating their Way through College,” a student remarked, “With ChatGPT, I can write an essay in two hours that normally takes 12.” Cheating is not new, but as one student exclaimed, “the ceiling has been blown off.” A professor remarks, “Massive numbers of students are going to emerge from university with degrees, and into the workforce, who are essentially illiterate.”

For Hu, removing AI from the equation does not negate cheating. Those who genuinely want to learn will choose how to use the tools wisely. Hu was at a recent panel discussion at Greenwich University and Hu commented to a question from a professor about whether to ban students from using AI: “Banning AI in education misses the point. AI can absolutely do good in education, but we need to find a way so students don’t offload their thinking to AI and lose the purpose of learning itself. The goal should be fostering critical thinking, not just policing the latest shortcut.”

Another professor posed the question, “If a student is not a native English speaker, but the exam requires them to write an essay in English, which approach is better?

• Allowing the student to write the essay in their own words, even if the English is poor, or
• Enabling the student to use AI to generate a well-written, grammatically correct essay by inputting their ideas and then copying and pasting the AI’s output?

Hu commented that not one professor on this panel could answer the question. The situation was unfathomable and far removed from situations covered by current policy and governance. She observes, “There is already a significant impact on education and many important decisions have yet to be made. It’s difficult to make clear choices right now because so much depends on how technology will evolve and how fast the government and schools can adapt.”

For educational institutions that have traditionally been centers of knowledge creation, the rise of AI is powerful — one that often feels more like a competitor than a tool. As a result, it has left schools struggling to determine how AI should be integrated to support student learning. Meanwhile, schools face a dilemma: many have been using generative AI to develop lessons, curricula, even review students' performance, yet the institution remains uncertain and inconsistent in their overall approach to AI.

On a broader scale, the incentive structures within education are evolving. The obsession with grades have “prevented teachers from using assessments that would support meaningful learning.” The shift towards learning and critical thinking may be the hope that students need to tackle an environment with pervasive AI.

MIT Study Sites Cognitive Decline with Increasing LLM Use

MIT Media Lab produced a recent study that monitored the brain activity of about 60 research subjects. These participants were asked to write essays on given topics and were split into three groups: 1) use LLM only 2) use traditional search engine only 3) use only their brain and no other external aid.

The conclusion: “LLM users showed significantly weaker neural connectivity, indicating lower cognitive effort and engagement compared to others.” Brain connectivity is scaled down with the amount of external support. This MIT brain scans show: Writing with Google dims your brain by up to 48%. ChatGPT pulls the plug, with 55% less neural connectivity. Some other findings:

• 83.3% of LLM users failed to provide even a single correct quote from their own writing.
• The word choice and the essay arguments were also much less diverse than the brain only group.
• Some students who used AI for essays didn’t claim ownership over their writing.

Hu noticed that the term “cognitive decline” was misleading since the study was conducted over a four-month period. We’ve yet to see the long-term effects. However, she acknowledges that in one study about how humans develop amnesia suggests just this: either we use it or lose it. She adds, “While there are also biological factors involved such as changes in brain proteins, reduced brain activity is thought to increase the risk of diseases that affect memory.”

The MIT study found that the brain-only group showed much more active brain waves compared to the search-only and LLM-only groups. In the latter two groups, participants relied on external sources for information. The search-only group still needed some topic understanding to look up information, and like using a calculator — you must understand its functions to get the right answer. In contrast, the LLM-only group simply had to remember the prompt used to generate the essay, with little to no actual cognitive processing involved. As Hu noted, “there was little mechanism formulating when only AI was used in writing an essay. This ease of using AI, just by inputting natural language, is what makes it dangerous in the long run.”

AI Won’t Replace Humans, but Humans using AI Will — is Bull S***!

Hu pointed to this phrase that has been circulating on the web: “AI won’t Replace Humans, but Humans using AI Will.” She argues that this kind of pressure will compel people to use AI, engineered from a position of fear explaining,

“If we refer to those studies on AI and critical thinking released last year, it is less about whether we use AI but more about our mindset, which determine how we interact with AI and what consequences you encounter.”

Hu pointed to a list of concepts she curated from various studies she called AI’s traits — how AI could impact our behavior:

1. People with low self-confidence will rely more on AI.
2. Those with poor learning skills will rely on AI in whatever activity they do.
3. Using AI for help will replace the support from friends.
4. AI has an 80% higher chance of being more persuasive than humans.
5. If people rely on AI for research or writing, without first using their own thinking they risk weakening their long-term memory and will feel less connected to their work.

Hu stresses that we need to be aware of these traits when we work with AI on a daily basis and be mindful that we maintain our own critical thinking. “Have a clear vision of what you’re trying to achieve and continue to interrogate output from AI,” she advises.

Shifting the Narrative So Humans are AI-Ready

Humanity is caught in a tug of war between the provocation to adopt or be left behind and the warning to minimize dependence on a system that is far from trustworthy.

When it comes to education, Hu in her analysis of the MIT study, advocates for delaying AI integration. First, invest in independent self-directed learning to build the capacity for critical thinking, memory retention, and cognitive engagement. Secondly, make concerted efforts to use AI as a supplement — not a substitute. Finally, teach students to be mindful of AI’s cognitive costs and lingering consequences. Encourage them to engage critically — knowing when to rely on AI and when to intervene with their own judgement.

She realizes, “In the education sector, there is a gap between the powerful tool and understanding how to properly leverage it. It’s important to develop policy that sets boundaries for both students and faculty for AI responsible use.”

Hu insists that implementing AI in the workforce needs to be done with tolerance and compassion. She points to a recent manifesto by Tobi Lütke’s Shopify CEO, that called for an immediate and universal AI adoption within the company — a new uncompromising standard for current and future employees. This memo shared AI will be the baseline for work integration, improving productivity, setting performance standards which mandates a total acceptance of the technology.

Hu worries that CEOs like Lütke are wielding AI to intimidate employees to work harder, or else! She alluded to one of the sections that demanded employees to demonstrate why a task could not be accomplished with AI before asking for more staff or budget as she asserts, “This manifesto is not about innovation at all. It feels threatening and if I were an employee of Shopify, I would be in constant fear of losing my job. That kind of speech is unnecessary.” Hu emphasized that this would only discourage employees further, and it would embolden CEOs to continue to push the narrative of how AI is inevitably going to drive layoffs.

She cautions CEOs to pursue an understanding of AI’s limitations for to ensure sustainable benefit for their organizations. She encourages CEOs to pursue a practical AI strategy that complements workforce adoption, considers current data gaps, systems, and cultural limitations that will have more sustainable payoffs. Many CEOs today may be more likely to pursue a message with AI, “we can achieve anything,” but this deviates from reality. Instead, develop transparent communication in lock-step with each AI implementation, that clarifies how AI will be leveraged to meet those goals, and what this will this mean for the organization.

Finally, for individuals, Hu advises, “To excel in a more pervasive world of AI, you need to clearly understand your personal goals and commit your effort to the more challenging ones requiring sustained mental effort. This is a significant step to start building the discipline and skills needed to succeed.” There was no mention, this time, of “AI” in Hu’s counsel. And rightly so — humans should own their efforts and outcomes. AI is a mere sidekick.

The above is the detailed content of Are We Finally Ceding Control To The Machine? The Human Costs Of AI Transformation. For more information, please follow other related articles on the PHP Chinese website!