Now loading...
A new study from Anthropic reveals a potential downside to relying on AI tools in the workplace: they might hinder skill development for software engineers. While previous research has shown that AI can accelerate certain tasks by up to 80 percent, this randomized controlled trial suggests that using AI to learn new coding concepts could lead to weaker comprehension. Involving 52 mostly junior developers, the experiment focused on mastering a Python library called Trio, which deals with asynchronous programming—a common professional challenge.
Participants were divided into two groups: one coding tasks manually and the other with access to an AI assistant capable of generating code. Both groups tackled two features using Trio, following a self-guided tutorial with problem descriptions and starter code. Afterward, they took a quiz testing four key areas: debugging errors, reading and understanding code, writing solutions, and grasping underlying concepts. The emphasis was on debugging, code reading, and conceptual knowledge, seen as vital for overseeing AI-generated outputs in real-world scenarios.
The results were telling. Those using AI finished about two minutes faster on average, but the difference fell short of statistical significance. More strikingly, their quiz scores averaged 50 percent, compared to 67 percent for the manual coding group—a gap equivalent to nearly two letter grades. The largest disparity appeared in debugging questions, raising concerns that AI could impair the ability to spot and fix flaws in code, especially if generated by machines.
Not everyone who used AI fared poorly; the approach made a big difference. Researchers analyzed screen recordings to categorize interaction styles. Low performers often delegated heavily to AI for code writing or debugging, showing less independent effort and scoring under 40 percent. For instance, some participants let the AI handle everything, completing tasks quickest but retaining little. Others started independently but shifted to full reliance midway.
In contrast, higher scorers—averaging 65 percent or better—integrated AI thoughtfully. Some generated code first, then queried the AI for explanations to build understanding. Others combined code requests with requests for breakdowns, spending extra time on these but gaining deeper insight. The top pattern involved posing conceptual questions to the AI, then applying that knowledge to code manually; this group resolved errors on their own and finished second-fastest overall.
These patterns, while not proving causation, highlight how passive AI use might encourage cognitive shortcuts at the expense of learning. The study underscores broader tensions in AI adoption. As coding tools automate routine work, engineers still need robust skills to guide, verify, and debug outputs, particularly in critical systems. For companies, this means rethinking AI policies to promote active engagement, perhaps through training on effective prompting or limiting over-reliance for novices.
The findings align with prior work showing reduced worker engagement and effort when AI assists, potentially stunting growth in fields like software engineering. Yet, they don’t negate AI’s productivity boosts for familiar tasks; instead, they warn that rushing new skill acquisition with AI could erode the expertise needed to sustain those gains long-term. Anthropic suggests designing AI products—like learning modes in tools such as Claude or ChatGPT—to support both speed and comprehension.
Limitations include the small sample and short-term quiz focus; longer studies could explore if effects fade with experience or extend to other professions. Still, the research calls for a balanced view: AI can transform work, but preserving human oversight requires deliberate strategies to nurture skills alongside efficiency. For the full details, see the paper on arXiv.
