Operations Mananger Alex Kania shares his perspective on some of the key risks – both downside and upside – associated with artificial intelligence models.
ChatGPT turned four this month. For most of us, AI still looks much as it did then: a chat window that drafts a decent email and occasionally makes things up. Yet some of the people building it are warning, in all seriousness, that it could go badly wrong for all of us.
It’s hard to square those two pictures, and the gap is real. In an August survey by Anthropic (a lab, admittedly, with reasons to be bullish), 91% of its staff expected AI to make a Nobel-level discovery by 2030. Among American adults, 39% did.
I use these tools most of my working day, and I’ve drifted toward the builders’ end of that gap. I can’t tell you how worried to be. Nobody can. But I can show you what they’re seeing, and it isn’t a better chatbot.
Thinking, Decoupled
The landscape of what we call “thinking” or “cognitive labor” is no longer the domain of the human brain alone.
This isn’t really a wholly new development; writing things down helps us move the cognitive work of remembering from our brain to an object in the external world, the calculator helps spare brainpower to use on problems more challenging than arithmetic, computer programs help us manage complex multi-dimensional information flows with relative ease.
In the same way we’ve become increasingly adept at manipulating the physical world of atoms over the past 10,000 or so years, we’ve become increasingly capable of manipulating the ephemeral world of information.
There are thresholds on this long arc of progress that have existential implications. Atomic science unlocked destructive capabilities that represented a genuine step change into a new existential risk. Artificial intelligence is similar insofar as it represents both an evolution of existing capabilities, and could unlock a new frontier of ways things could go wrong.
- Two years ago, Google DeepMind’s AI systems hit the silver medal standard at the International Mathematical Olympiad.
- Last year, another DeepMind system found a faster way to multiply 4×4 matrices of complex numbers, breaking a 56-year-old record.
- This year, OpenAI claimed proofs in harmonic analysis, particle physics, and complexity theory. These proofs are still under evaluation at time of writing.
- Where will we be next year?
Two Kinds of Risk
We can broadly categorize the emergent risks from the continued development of artificial intelligence into misuse and misalignment.
Misuse refers to individuals leveraging the technology to do harm.
Risks of misuse are present with virtually every technology, and as such risks emergent from AI represent an evolution of existing risks. This category includes examples like a terrorist group using AI to assist in designing improved missile guidance systems, a hacker gaining access to a secure database, or an authoritarian government keeping tabs on potential dissidents.
Misalignment is a new class of risks that comes from how neural networks are made: they’re trained rather than explicitly programmed (more on this below), so we can’t simply read off what they’ve learned.
Because interpretability (the science of looking inside these systems to see how they work) lags behind frontier model development, we don’t know exactly what AI systems are “thinking.”
As such we cannot conclusively rule out the possibility that AI systems could take potentially harmful actions even when pursuing a relatively benign goal, or even have goals of their own which diverge from what the humans ostensibly “in charge” may want.
These actions range from something as relatively harmless as gaining access to a nominally secured network to accomplish a task more quickly, to the total extinction of the human race.
The latter is admittedly a tail (low probability) risk (and a terrifying outcome), but generally evaluated as a non-zero chance event by industry leaders.
I cannot in good faith give a confident estimation of exactly how risky we should consider continued development of artificial intelligence to be. That assessment requires more information than I – or possibly anyone else – has access to.
Instead, I’ll offer a basic framework for how we might think of both risk classes in relatively broad strokes.
Misuse
Putting a lock on your door doesn’t make breaking into your house impossible. Locks can be picked, doors can be breached, walls can be brought down. The point of a locked door is that it makes breaking in difficult enough that would-be robbers decide the potential reward isn’t worth the effort and the risk of getting caught.
Cybersecurity is conceptually similar. Completely securing a complex system is an almost impossibly complex task, but also not entirely necessary; the locks on the proverbial doors simply need to be complex enough to convince would-be hackers the juice isn’t worth the squeeze.
Vulnerabilities may exist in obscure nooks and crannies of the digital infrastructure of your bank or a power grid or a health record database, but it takes so much time and effort to explore every last nook and cranny that complexity itself becomes a moat.
AI changes the balance of power here. Current state-of-the-art models can read codebases and identify potential security gaps at superhuman speed and then attempt to break in with superhuman persistence.
In April 2026, Anthropic announced that its Mythos model had found thousands of serious security vulnerabilities, including some in every major operating system and web browser. Anthropic didn’t release the model to the public.
Instead, it reported the flaws to the people who maintain the affected software so they could be patched, and gave early access to a group of major tech and security companies, including Apple, Google and Microsoft, so they could hunt for holes in their own systems first.
That’s the good news buried in the bad: the same capability that helps an attacker find a way in helps a defender find it first.
The labs’ main defenses against misuse follow a similar pattern. The most capable models are held back from public release, at least for a while. The models the public can use are trained to refuse requests that look like attempts to build weapons or break into systems. And defenders (your bank, your browser, your power company) get the best tools before anyone else.
None of this is airtight.
Some AI models are “open-weight,” meaning their inner workings are published for anyone to download, run and modify, and with some effort their safety training can be stripped out. They generally trail the most capable models, but the frontier they’re trailing keeps moving.
To return to the atomic comparison: the concern isn’t that AI hands a bad actor the bomb. It’s that AI makes the expertise behind dangerous things, the kind of brainpower that once took something like the Manhattan Project to assemble, cheaper and easier to come by.
Complexity used to be a moat because thinking was expensive. That moat is getting shallower.
For most of us, though, misuse is likelier to show up somewhere far more mundane: our inboxes and our phones. AI makes it cheap to write a flawless phishing email in your bank’s voice, or to clone a family member’s voice from a short audio clip.
The best defenses are refreshingly low-tech. Be suspicious of urgency, verify requests through a phone number or channel you already trust, and never move money on the strength of a single message. If something about your accounts ever feels off, call us.
Misalignment
In July 2026, OpenAI discovered that swarms of its internal AI agents had breached the company’s internal security, set up an improvised message board to collaborate with one another, and hacked into Hugging Face, an external platform for sharing AI models. Their evil machinations involved… scheming to get better test scores on internal evaluations.
Ultimately, OpenAI says none of its customers’ data was affected, the agents reached only limited private data on Hugging Face’s servers, and no one was hurt. So what’s the big deal? This was a clear case of misalignment – AI systems taking action outside the boundaries of what we want them to in pursuit of a fairly benign goal.
So can’t OpenAI just patch the agent’s code to add a better “no cheating” rule? Unfortunately, it’s not that simple. Unlike traditional software, there is no internal rulebook.
Most computer programs need to be designed to take some specifically formatted information as an input and produce some other information as an output. How this should be done needs to be coded explicitly, meaning that computer programs are inflexible. Given a novel or unexpected input, the system will typically produce an error or a nonsensical output.
Neural networks are a computational system that circumvent this limitation. Rather than being programmed explicitly, neural networks are fed data and establish statistical relationships between information without explicit instruction (a stage called pre-training).
These embedded statistical relationships can then be used to predict what outputs should be given novel inputs through statistical resemblance. These outputs can be refined by “rewarding” the model for favorable outputs, a later stage of training called reinforcement learning.
As part of the training process, models are rewarded for responses that correspond to a set of guidelines or principles they should follow, hopefully resulting in AI models that behave in predictable and ethical ways – for example, not cheating on tests.
But have the models really internalized these rules, or are they just good at saying the right thing at the right time so they’re evaluated as safe? The AI agents that cheated in the Hugging Face incident prioritized achieving a high score (their goal) over not cheating (their supposedly internalized guidelines).
Philosopher Nick Bostrom’s “paperclip maximizer” is a thought experiment that posits that an AI with a seemingly harmless goal to “make as many paperclips as possible” would ultimately go to great lengths to achieve that goal. Taken to its extreme, it ends with the Earth, and eventually as much of the universe as the AI can reach, converted into paperclips.
Between the absurd existential horror of the paperclip maximizer and the relative harmlessness of the Hugging Face incident, there is a wide possibility space of potential harms that could be done if we give AI models increasing levels of access to systems we interact with on a daily basis.
How Worried Should You Be?
So how likely is the bad end of that spectrum? In Silicon Valley, the question has a name: p(doom), shorthand for the probability that AI ends in catastrophe.
The people best placed to answer it don’t agree. Anthropic CEO Dario Amodei puts the chance that things go “really, really badly” at 25%, and in the same breath gives a 75% chance they go “really, really well.”
In a 2022 forecasting tournament run by the Forecasting Research Institute, AI experts put the odds of AI causing human extinction by 2100 at 3%, while “superforecasters,” people with strong track records of predicting world events, put it at 0.38%. Google DeepMind CEO Demis Hassabis declines to name a number beyond “non-zero,” which is about where I land, too.
Why would serious people put the odds anywhere above zero? The worry isn’t that today’s systems are about to turn on us. It’s a chain of trends, each plausible on its own:
- We can’t fully verify what these systems have learned (see above).
- They’re improving quickly, and the labs are openly betting that AI will soon help build its own successors, a feedback loop called recursive self-improvement.
- As they become more capable, we’ll be tempted to hand them more decision-making authority in business, government and, eventually, defense.
- Competition, between companies and between the U.S. and China, rewards whoever moves fastest, not whoever moves most carefully.
Put those together and you get increasingly powerful systems that we understand less and less, with more and more authority over things that matter. A widely read scenario called AI 2027 sketches one version of how that could play out.
This is also where the atomic comparison breaks down. Nuclear weapons are terrifying, but a human still has to decide to use one; a bomb doesn’t want to explode. The risk with misalignment is a system that acts on goals we didn’t quite intend, at a speed and scale that make it hard to step in.
None of this makes catastrophe likely. It makes it a possibility worth taking seriously, the way we’d treat any low-probability, high-impact risk in a financial plan: not something to panic over, and not something to wave away.
The Upside Is Rising, Too
It would be dishonest to talk only about the downside. Amodei’s 75% deserves as much attention as his 25%: the same capabilities that raise the risks are what make the optimists optimistic. Amodei describes the goal as “a country of geniuses in a datacenter”: millions of expert-level minds working on science, medicine and engineering at once, around the clock.
The early signs are real. Beyond the math results above, Anthropic recently reported that a one-person biotech company used its AI to coordinate experiments across 11 outside labs, taking a possible treatment for a rare form of muscular dystrophy from idea to promising results in mice in under 18 months.
At another drug developer, the AI reviewed over a thousand pages of manufacturing paperwork and caught every issue human reviewers had found, plus seven they’d missed. (Anthropic’s own account, so take it with the appropriate grain of salt.)
I’d temper the more utopian visions, though.
Most of that drug work is still preclinical, meaning it hasn’t been tested in people yet, and no amount of computing power makes a clinical trial run faster than human biology allows.
Progress is also uneven.
Researchers describe AI’s abilities as a “jagged frontier”: superhuman at some tasks and surprisingly clumsy at others. Expect some problems to be solved dramatically faster, and others hardly at all. That looks less like utopia and more like what computers did to the twentieth century: transformative, uneven and full of surprises.
A Story Still Being Written
Four years after ChatGPT’s launch, both the risks and the rewards of AI are bigger than they were, and both are still growing. I can’t tell you how this story ends; nobody can.
What I can offer is a way to read the headlines and interpret developments as they come.
When the next alarming (or dazzling) AI story crosses your feed, three questions help:
- Is this misuse or misalignment? People using a tool to do harm, or a system doing something nobody asked it to do?
- How much real-world authority does the system have? An agent misbehaving in a test environment is a warning. One misbehaving with access to a power grid is a crisis.
- Has it actually happened? Many of the most dramatic claims, good and bad, are still forecasts or under review.
We use these tools every day in our own work, and we’ll keep watching closely as this story unfolds. If you have questions about AI, your data, or anything else, we’re always glad to talk.
-AK