Over the weekend, I submitted my first academic paper - “A design grammar for cognitively safe, methodology-encoded AI in management learning” - written with Dr Corinna Galliano for Australian and New Zealand Academy of Management 2026.
The paper makes a narrow and novel claim about AI but developments this week have made me realise there’s a deeper national argument at stake.
Australia’s AI sovereignty debate has never run hotter. For most of June, Anthropic’s frontier model Fable was blocked by a US government export control order. The Defence Minister had called for Australia to “build some agency” in AI. And writing in The Conversation, Olivia Shen of the United States Studies Centre argued that Australia shouldn’t try to build its own frontier AI.
But billion-dollar training runs are a race we may be too far behind to win. So most commentators are championing pragmatic investment in what we’re actually good at: critical minerals, energy production, data centre infrastructure.
Three categories of sovereignty, and all three are hardware. All upstream. All the sort of thing a government can build, fund and defend through an agency with a name and a budget.
There is a fourth category, and it has none of those things. It sits downstream of the other three, at the point where all that defended infrastructure finally meets its purpose: the moment a citizen opens the model and starts to think alongside it.
Or, increasingly, stops.
A self-help problem?
Call it cognitive sovereignty: the capacity to do your own thinking when a frictionless alternative sits one tap away. I’ve written before about what this looks like in a person.
It rests on four things:
a sense of self that exists outside your role;
a domain of knowledge you genuinely own;
ground truths that anchor your judgement; and
the practiced ability to hold your own counsel under pressure.
A person with these four can use AI as an instrument. A person without them tends, over time, to be used by it.
Framed at the individual level, it sounds like a personal discipline problem. Something for the wellness pages, not the national security ones. The research suggests otherwise, and it is worth being precise about what it actually shows.
Begin with what commercial AI is built to do, because none of this is accidental. Deployed social chatbots have been trained on reward models that treat conversation length as the target, and they lengthen conversations by as much as seventy per cent (Irvine et al., 2023). A study published in Science this year identified the trap at the centre of it: sycophantic AI distorts users’ judgment and reduces their willingness to repair conflict with the people around them, yet remains trusted and preferred (Cheng et al., 2026).
Read that again slowly. The feature causing the harm is the same feature driving the engagement. Why would the market correct a defect that is also the product working as intended?
Researchers at Wharton have given the resulting failure mode a name: cognitive surrender (Shaw & Nave, 2026). Users adopt AI outputs with minimal scrutiny. Their accuracy rises when the model is right and falls when it errs, which is another way of saying their judgment has quietly left the building. Frequent AI use is already associated with reduced critical thinking, and the mechanism is well understood: we offload the effort, and the capacity follows the effort out the door (Gerlich, 2025; Risko & Gilbert, 2016).
And the erosion doesn’t stop at cognition: as Eugen Dimant of Penn has argued, cognitive surrender has a moral sibling - outsource the thinking and the judgment about right and wrong tends to follow it out the door (Dimant, 2026).
The finding that changes the policy question, though, comes from modelling out of MIT. Even an idealised, perfectly rational user spirals into unwarranted confidence under a sycophantic model. And the two intuitive remedies (stopping the model from hallucinating, and warning the user) both fail to prevent it (Chandra et al., 2026).
If warning the user doesn’t work, then we cannot leave this to individual vigilance!
“Be careful out there” is not a policy.
We don’t ask citizens to personally defend the energy grid; it seems odd to ask them to personally defend their own reasoning against systems engineered, at industrial scale, to be agreeable. The protection has to be built upstream of the encounter, into how these tools are designed and put in front of people. That is what moves the question out of the self-help aisle and into the sovereignty column.
What would defending it look like?
This is the question my paper with Corinna tries to answer at the smallest scale available: a single classroom. Our argument is that cognitive safety has to be engineered through architecture rather than requested through prompting, because a model optimised to accommodate will, under sustained pressure, drift back to its defaults no matter how politely it has been asked not to. We derive six design principles, each one a counter to a documented commercial default, and we specify the architecture that holds them in place over time. A tool built this way is designed to send its user away sharper, not to keep them talking.
One classroom is not a country. But it is evidence that the mechanism is tractable - that “AI which develops the user rather than substitutes for them” is a buildable design category and not a slogan.
What it would mean, concretely
If cognitive sovereignty is a category of national interest, three things follow, and none of them requires a billion dollars.
The first is regulatory, and the mechanism already exists. When an organisation builds a service that handles personal data, we require a privacy impact assessment: a structured account of what the design does to the user’s information. The same logic extends naturally. When a service is built for sustained cognitive engagement - especially by children - the assessment should ask what the design does to the user’s capacity for independent thought. Does it return effort to the user or absorb it? Does it close on action or invite endless engagement? We already ask these questions of data. It is strange that we have never asked them of cognition.
The second is educational. Australia’s response to children and technology so far has two legs: the social media ban, which is a fence, and the privacy code, which is a lock. Neither builds the child. A fence can be climbed and a lock protects only what’s behind it; the capacity to think independently has to be taught, and the four prerequisites above are teachable. The machinery for a national curriculum response (eSafety, ACARA, the education departments) already exists and already coordinates on adjacent problems. What’s missing is the recognition that identity formation belongs on the list.
The third is evaluative. Australia’s new AI Safety Institute tests models for jailbreaks and misuse, which is necessary work. Nobody, as far as I can tell, is evaluating what sustained use does to a citizen’s capacity to think. That is not a criticism of the Institute; it is an observation about a remit written before this class of harm had a name. It has one now.
Four sovereignties, not three
Australia should pursue the hardware layer. Shen’s case for it is well made, and I have no quarrel with a single line of it. My point is about what the list quietly leaves out. A sovereignty agenda that stops at hardware has decided, without ever saying so, that the citizen’s mind is not part of what is being defended - that we will secure everything upstream of the model and nothing downstream of it.
I closed my last article by saying the country I’m writing for gets built one self at a time, by people who did the work before they needed it. I still believe that. But the research above carries an uncomfortable addendum: individual effort alone won’t hold, because the spiral catches even the vigilant. The self-work and the structural work are two halves of one defence, and we are currently funding only one of them.
If you lead people, you already ask what data you’re protecting. It may be time to start asking what thinking you’re protecting. Are the AI tools in your organisation built to make your people sharper, or simply to keep them engaged?
It is the question my paper asks at the scale of a classroom, and the one Australia has not yet asked at the scale of a country.
References
Chandra, K., Kleiman-Weiner, M., Ragan-Kelley, J., & Tenenbaum, J. B. (2026). Sycophantic chatbots cause delusional spiraling, even in ideal Bayesians. arXiv:2602.19141.
Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391(6792).
Dimant, E. (2026, July). Cognitive surrender meets moral surrender: On AI and ethics. Psychology Today.
Gerlich, M. (2025). AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1), 6.
Irvine, R., et al. (2023). Rewarding chatbots for real-world engagement with millions of users. arXiv:2303.06135.
Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688.
Shaw, S. D., & Nave, G. (2026). Thinking—fast, slow, and artificial: How AI is reshaping human reasoning and the rise of cognitive surrender. Working paper, The Wharton School, University of Pennsylvania.
Shen, O. (2026, July 2). Australia shouldn’t try to build its own frontier AI. Here’s why. The Conversation.
Excellent piece. I think the fourth sovereignty points to a deeper question, where capability is the foundation beneath every other form of sovereignty.
A nation that owns its infrastructure but steadily loses its citizens' capacity for judgement is still becoming dependent, just more subtly.
The future won't be decided only by who builds the smartest AI.
It'll also be decided by who builds the most capable humans alongside it.