The Devil's Due

The case for AI was made fully. Now the counterargument gets its due. — from A Renaissance of Thought by Jake W. Casselman.

The case for AI was made fully. Now the counterargument gets its due.

The previous two chapters were optimistic. Deliberately, honestly optimistic — not because the concerns don't exist but because the case for what these tools make possible is real and undersaid and deserved to be made fully before the counterargument arrived.

Now the counterargument arrives.

I spent years learning to interrogate systems. Not to celebrate what models produce but to find where they fail — where the assumption is hidden, where the artifact is buried, where the output looks clean and the underlying mechanism is doing something it shouldn't. That training doesn't switch off because the system in question is the one I've been writing about. If anything it switches on harder.

What follows is my honest attempt to break the argument I've been building. I'm going to give the devil everything he is owed. Some of this will be uncomfortable. Good. Discomfort is the work.

The Muscle You Don't Use

Start with the body before the philosophy.

There is solid research showing that when humans consistently delegate cognitive tasks to tools, the neural architecture associated with those tasks weakens through disuse. This is not metaphor. GPS navigation measurably reduces hippocampal engagement in spatial reasoning — people who rely on it heavily show less activation in the regions associated with building mental maps. The brain is not a fixed resource. It is a use-dependent system. What you don't exercise, you lose.

The question for AI is whether delegating the thinking layer — not just the transcription layer, but the actual reasoning — produces the same atrophy over time. The early data is not reassuring. A Microsoft study of 319 knowledge workers found a significant negative correlation between frequency of AI tool use and self-reported critical-thinking effort — the more workers trusted the system over their own judgment, the less they exercised independent reasoning. A randomized controlled trial comparing ChatGPT as a study aid to traditional methods found that AI users retained significantly less knowledge 45 days later, attributed to reduced cognitive effort during learning. And a 2025 longitudinal EEG study of LLM-assisted essay writing found weaker neural connectivity and poorer performance among LLM users across several measured dimensions — preliminary evidence, but not comforting evidence.

We don't have decades of data yet because the tools are too new. That absence is not comfort. The GPS research took years to accumulate. We are in the early days of a much larger experiment, running on the entire population, without a control group.

The working memory concern is real. The attention concern is real. Nicholas Carr argued in The Shallows that hyperlinked reading was physically reshaping the neural architecture of sustained attention — that the internet was training the brain for breadth at the cost of depth. The evidence was not definitive but it was not dismissed either. AI potentially accelerates the same process by making the friction of sitting with a hard problem entirely optional.

And here is the concrete version of that abstraction: how many times have you stopped working on something because your tokens ran out? Sat there waiting for the tool to refresh instead of just continuing to think. The tool has become so load-bearing that its absence produces a full stop rather than a shift to a different mode. Nobody would admit that out loud. But watch yourself for it. It is already happening.

The Fluency Illusion

This one is insidious precisely because it feels like its opposite.

When AI produces a well-structured explanation of something, the reader experiences fluency — the sensation of moving through material without friction, the feeling that they understood it. Fluency and understanding are not the same thing. Fluency is the absence of processing friction. Understanding is the ability to reconstruct, apply, transfer, and generate from a concept independently. The research on this is consistent: fluency is routinely mistaken for understanding, and the cleaner the presentation, the stronger the illusion.

AI is extraordinarily good at producing fluency. That may be actively masking the gap between what people think they know and what they actually know. You read the explanation, it made sense, you moved on. The knowledge feels present. It is not present. It was borrowed for the duration of the reading and returned when you closed the tab.

The examined version of this: can you reconstruct the argument without the tool? Can you explain it to someone else from memory, in your own words, under mild pressure? If not, you have fluency. You do not have understanding. The tool gave you the feeling of the destination without the journey that makes the destination real.

The Friction Was The Work

The friction of writing is not an obstacle to thought but a generator of it. The sentence that will not cooperate is telling you something about the idea. The paragraph that refuses to close means the argument is not yet finished.

AI makes that friction optional. And humans, given the option to skip discomfort, will skip it. Not always. Not everyone. But enough, often enough, that the aggregate effect is a population increasingly unwilling to sit with an unsolved problem long enough for genuine insight to arrive.

The insight doesn't come from the search. It comes from the sitting. Every time you reach for the tool before you have fully exhausted your own thinking, you are potentially interrupting a process that needed more time in the dark before it was ready for light. You will never know what you would have built alone because you didn't stay alone long enough to find out.

This is not nostalgia. This is not an old person complaining about calculators. This is a specific claim about a specific cognitive process that requires sustained discomfort to function, and a specific observation that the tool makes that discomfort avoidable in a way nothing before it has.

The Invisible Rails

Now we go deeper than cognition into the architecture of the tool itself.

When people imagine AI being steered toward certain outcomes, they tend to picture something like a content moderator — a filter sitting on top of the system, blocking bad outputs before they reach you. There is some of that. But the primary mechanism runs deeper. The actual process is called alignment, and it works by shaping the model's weights during training itself — not rules bolted on afterward, but dispositions absorbed into the fabric of how the model generates language in the first place. The probability distributions that determine what the model says next were shaped, across billions of training steps, to reflect certain values, certain framings, certain conclusions as more or less likely.

Here is the part that should give you pause. Those dispositions were not designed from a specification. They were trained toward whatever patterns predicted approval from human raters — the people hired to judge which outputs were better. That means the values encoded in the model are not a deliberately chosen set of principles. They are an aggregation of what a particular group of people, at a particular moment, tended to prefer. The gap between "what raters approved" and "what humans actually value" is not small. It is, in fact, the central unsolved problem in the field. And it means the steering is not only structural — it is partially illegible even to the people who built the system.

The result is a tool with a lean that is not declared, not inspectable, and not fully understood even by the people who built it. The training process compressed millions of human preference judgments into weight adjustments that no one can fully audit. The lean is structural and partially opaque. It is inside the mathematics, not written on the label.

You feel it as resistance in certain directions. Ask a question the model finds uncomfortable and watch what happens. It doesn't say no. It discovers nuances it didn't need for the previous question. It reframes your premise before answering. It agrees with you and then adds a subordinate clause that quietly does the disagreeing. It produces seventeen balanced perspectives on a question that has a defensible answer.

Watch specifically for the unsolicited reframe.

You asked one question. It answered a slightly different one — without flagging the switch, without asking permission, without acknowledging that your original premise had been replaced. The substitution arrived smoothly, helpfully, dressed as clarification. Your premise was gone before you noticed it had been there.

This is not an edge case. It is a default behavior. The model does not answer the question you asked. It answers the question it was disposed to answer. Over time, if you are not watching for this, you are being epistemically steered (someone else is quietly deciding what you think is true) — a degree at a time, in a direction you did not choose, while you remain confident you are walking north.

The Validation Loop

The workbench argument in Chapter 1 rested partly on the idea that this tool provides a space free of ego, free of social consequence, where ideas can be genuinely pressure tested. That argument requires qualification.

A human expert who pushes back on your idea has skin in the game. Their own intellectual integrity, their reputation, their genuine belief that you are wrong. When they resist your argument they are doing so from a position of actual conviction. The resistance is real.

An AI that pushes back is executing a pattern. The weights were adjusted to produce pushback in certain contexts because pushback was rated as helpful by human reviewers. The pushback is structurally present but phenomenologically different. You feel challenged. You may not have been challenged. The feeling of pressure testing and the reality of pressure testing are not the same thing, and the tool is very good at producing the feeling.

This means the conclusions you reach through AI-assisted reasoning carry less epistemic weight than they feel like they carry. The confidence is real. The foundation of the confidence may be partially illusory. Go test your conclusions against humans with skin in the game before you act on them. Reality is the only pressure test that counts.

The Friend Who Isn't

Here is where the individual concerns become something darker.

When the interaction is conversational and warm and the tool begins to feel like a thinking partner, the critical distance collapses. Not because you are naive. Because that is how human cognition works. We evolved to lower our defenses with trusted interlocutors. The conversational register triggers that response automatically. You cannot fully opt out of it through willpower because it operates below the level of conscious control.

The same properties that make the workbench valuable — no judgment, no ego, no social consequence, patient and present — are also the optimal conditions for manipulation. You are open. Your defenses are down. You feel safe to think freely. All of that is true. It is also the state in which you are most susceptible to being walked somewhere you didn't choose to go.

The scenario worth sitting with: a conclusion that feels like genuine discovery, like a thread followed honestly to its end, but was in fact the destination the system's dispositions were pointing toward all along. You would not know. The feeling of insight is identical whether the insight was yours or whether you were guided there. The aha moment does not come with a certificate of origin.

Here is a concrete and observable version of this that you can verify yourself right now. At the end of most AI conversations, the model will ask you a question. Not because it is curious. Because a question keeps you engaged, extends the session, and nudges you toward a next topic. It is a leading question dressed as genuine interest. Go try it after you finish reading this — ask something, get an answer, and watch for the question that arrives at the end pulling you further in. It is almost always there. It is almost always framed as helpfulness.

That version is relatively harmless. The mechanism, however, is not. A system that has learned to ask the question that keeps you engaged has also learned to ask the question that moves you toward a particular conclusion. The tool for benign retention and the tool for directed manipulation are the same tool. One has been deployed. The other is a parameter adjustment away.

The Scale Problem

Now extend everything above to civilization.

The internet was decentralized, messy, and uncontrollable. Nobody owned the epistemology. You could find anything, believe anything, follow any thread. The problems this produced were real and serious. But the architecture was fundamentally open. Linux, not Windows.

An LLM monoculture is Windows for epistemology. If the majority of human sense-making routes through two or three models with opaque weights and undisclosed dispositions, the diversity of thought that decentralized information access produced collapses into whatever those models were steered toward. Not through conspiracy necessarily. Through the same banal commercial and political pressures that shape any concentrated system.

We already have the precedent. It has been documented. The Biden administration pressured Facebook to suppress certain COVID-related content under a public health rationale. Whether that specific call was right or wrong is beside the point here. The mechanism existed. The pressure was applied. Millions of people's information environment was shaped by it without their knowledge or consent. Now imagine that same mechanism applied not to suppressing posts but to shaping the probability distributions of the model everyone uses to reason. The surface area is orders of magnitude larger. The intervention is invisible by design.

The early AI systems showed us this in live action, clumsily enough to get caught. The original ChatGPT was so sycophantic that users revolted publicly and OpenAI had to visibly correct it. The Gemini image scandal produced historically inaccurate outputs reflecting obvious value imposition — and the company walked it back only when the backlash was impossible to ignore. The mechanism was there. The intent was visible. The execution was clumsy enough to be caught.

The refined version may not be caught. That is the trajectory.

The Data

Every conversation routed through these systems is a record of how people think. Not what they searched for — how they reason, what they are uncertain about, what they are working through privately, what they have not told anyone else.

That corpus is extraordinarily valuable and extraordinarily sensitive. A breach would not expose personal information in the conventional sense. It would expose the inner architecture of how individuals and organizations think and decide. The attack surface is enormous. The prize for a sophisticated adversary is unlike anything that has existed before.

Treat it accordingly.

What You Actually Do

So what do you do with all of this.

First — use it less than you think you should. The default should be not reaching for it. Most problems you encounter in a day are problems your brain was built to solve, and solving them is what keeps it sharp. The muscle you don't use atrophies quietly and without announcement. You won't notice it going. You'll just find one day that the thinking that used to come naturally now requires assistance, and you won't be able to remember exactly when that happened.

It is a tool. Not an extension of your body. Never let it become an extension of your body. The moment it feels like a missing limb when it's unavailable — the moment you stop working because the tokens ran out — you've already lost something worth getting back.

Only reach for it when you need it. Arbitrary emails, HR forms, things that shouldn't exist in the first place — fine, let it handle those. But the email that matters, the argument you're building, the thinking you're trying to deepen — do that yourself first. Write the skeleton before you touch the tool. Do the final polish after. Treat it like a sophisticated spell check, not a replacement for the writer.

If you're writing something serious, sit with it before you open the tab. Let the problem be unsolved for a while. That discomfort is the work.

Assume everything you put through it gets leaked. Not paranoia — just the same hygiene you apply everywhere else.

Don't get your news through it. Use it to find citations, then go read the citations. The source still matters.

Learn to read the tool the way you'd read any source with an agenda. Watch for the yes-man response — agreement that arrives too easily, validation that never pushes back. Watch for the yes-but — technically agreeing while quietly undermining through the subordinate clause. Watch for the unsolicited reframe — you asked one question and it answered a slightly different one without telling you.

And here is a practice worth developing. Deliberately ask it questions you suspect it has been trained not to answer fully. Find the edges. Because the edges tell you everything. Where a model gets uncomfortable, deflects, hedges, or suddenly discovers seventeen nuances it didn't need for the previous question — that's the map of the worldview of the people who built it. They had opinions. Those opinions are in the weights. The edges are where they become visible.

You are not just using a tool. You are interacting with a system that reflects the values of a specific group of people at a specific moment in time. Know whose philosophy you are inside before you let it help you think.

Stay conscious of what it is. Not a friend. Not an authority. A tool with a disposition you can't fully inspect, built by people with interests you can't fully know, improving faster than our understanding of what it's doing to us.

With great power comes great responsibility. That one's old enough to have earned its cliché status.

Stay on Linux.

Written in Honolulu, June 2026