In less than 48 hours, a resignation notice published on X by a former Anthropic researcher, Jacob Coxon, has received more than 148 million views.
The post features a dire warning: “[T]he people building AI earnestly believe that it could kill us all by the end of the decade.”
A little over an hour after Coxon’s post appeared, a current Anthropic safety executive, Evan Hubinger, provided this response: “Jacob is correct here--we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
This is not the first time this has happened. In February, another Anthropic researcher, Mrinank Sharma, wrote: “The world is in peril. And not just from Al, or bioweapons, but from a whole series of interconnected crises unfolding in this very moment.”
Coxon’s letter has generated an explosion of global media coverage. Using Ground News, I scanned through more than 540 news headlines. All of them highlight the risk, I haven’t yet found one telling us what to do about it.
Is Civilization At Risk?
A reasonable question to ask is: Is civilization actually at risk from AI right now? Anthropic’s answer: The risk is low but accelerating.
In his post, Hubinger linked out to Anthropic’s latest Risk Report, a 180+ page document that looked at three categories of risk:
“An AI model with access to powerful affordances within an organization could use its affordances to autonomously exploit, manipulate, or tamper
with that organization’s systems or decision-making”: Anthropic ranks the risk of its own models doing this as low. However, the recent Hugging Face attack, where 100s of agents colluded, broke out of containment and hacked Hugging Face’s internal systems, suggests this risk is elevated.
“Highly capable AI models may be able to perform automated research and development (R&D) that rapidly accelerates progress in technical fields. If under human control, this acceleration could disrupt the balance of power both within and between nations; if combined with dangerous autonomous goals from AI, this could lead to catastrophic harms initiated by AI systems themselves.”: Anthropic says the risk is low. However, it admits that it can no longer reliably measure this risk because its evaluations cannot “capture increases in models’ capabilities” in this area.
“Individuals or small groups with limited resources use AI models to gain access to non-novel chemical or biological (CB) weapons.” Anthropic says the risk is low but has increased since its last report. One worry is that open source models with similar capabilities might aid the development of bioweapons. In addition, the creation of synthetic viruses with AI assistance is raising further concerns in this area.
What Should We Be Worried About?
So Anthropic is saying we shouldn’t be overly worried that AI will end civilization -- yet. The real threat is self-improving super intelligence that evolves rapidly and cannot be monitored.
We may be inching closer to this threshold. Open AI’s recently released Astra model is one example. Traditionally, models have explicitly generated thinking Chain of Thought (CoT) tokens (e.g., “Let us breakdown this problem into steps”) that can be observed to understand the model’s thinking process.
AI Agents We Can’t Monitor
Astra uses a different technique where it generates tokens inside a hidden latent space. Importantly, this may mean that some of the model’s thinking process is also hidden. From a safety perspective, this matters because if we cannot observe how LLMs are making decisions, it is harder to understand what they might do, and determine whether training efforts are properly aligning the model from a safety perspective.
In the Astra safety overview, OpenAI says “We have found that GPT‑6 Astra is more capable of controlling its own CoT than GPT‑5.6 Sol, and less likely to include incriminating information in its CoT ... These findings indicate that the Astra class models could evade our CoT monitors under adversarial conditions.” Translation: It’s harder to monitor Astra, in certain situations. If we can’t monitor the model, how can we verify it’s not going to act against our self-interest?
Overall, the risk that AI can currently give anyone the ability to manufacture bioweapons, hack into systems at will or take over a country’s nuclear arsenal is low. That’s great. The time to take these issues seriously is yesterday.
But the immediate threats to your personal and professional well-being from AI are much higher, and are already present. Here are three risks you should pay attention to, along with practical guidance you can use to protect yourself.
Three Personal and Professional AI Threats: What They Are and How to Stop Them
The best way to understand AI risk is to consider the question: “What pre-existing risks could AI rapidly accelerate from yellow to red alert, and what does that mean to me?” There are many, but here are three that you may not be paying enough attention to, and the risk level associated with each.
Data Privacy: AI Labs Have Your Data, What Will They Do With It?
Risk Threat Level: Elevated
AI systems already have access to huge amounts of personal and professional data. In some cases that information is being shared with third parties or analyzed internally for a variety of purposes. AI’s data footprint is increasing and autonomous agents (and even AI labs) are starting to act on this information. AI labs won’t steal your data, but it may be used to train models.
Recently, OpenAI launched software that lets ChatGPT access a user’s messages. ChatGPT can find information about meetings, family events, or even remind you about urgent messaged. It’s a potentially helpful feature that Proton calls a “privacy backdoor.”
Specifically iMessage “uses end-to-end encryption, which means only the sender and recipient can read the contents of a conversation ... The Apple Messages plugin for ChatGPT changes [this] ... If you authorize ChatGPT to scan iMessage conversations, those contents enter the AI pipeline and are treated like any other conversations in accordance with the ChatGPT privacy policy.”
That means any messages ChatGPT accesses could be sent to third parties (such as law enforcement) or be used to train OpenAI’s models.
The most important step you can take to protect yourself is to think carefully about giving ChatGPT, Claude and other AI labs blanket access to your computer files, financial records and other sensitive information on your computer or other devices.
Currently, there’s a brewing controversy associated with the recent solving of a long-standing math problem. The mathematician who solved the problem, Tristan Buckmaster, has accused OpenAI of using prompts he sent to Codex to train the model it used to develop its own mathematical proof. OpenAI said: ““While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models. However, our proofs differ significantly and even the precise results proved are different in the Euler case (forced vs unforced).”
The lesson: Consider any information you send to a non-local model as not private by default and accessible by third parties.
Digital Security: AI is Helping Hackers Move Faster
Risk Threat Level: Elevated
AI is accelerating the use of deep fakes, no-click malware that can infect mobile devices instantly, and efforts to infiltrate systems that process your payments, or the financial infrastructure your business depends on. One example is the VoidLink malware targeting Linux systems, which is “one of the first instances of an advanced malware largely generated by AI.”
In 2024, “a finance worker at engineering firm Arup authorized $25.6 million in wire transfers to accounts controlled entirely by AI-generated impersonators; every person on the video call was a deepfake.”
It’s easy to get spooked by growing digital threats. Are risks, like an AI hacking your smart fridge, remote? Likely yes. The more serious risks are to your personal digital safety. The best protection is to:
Keep your devices up-to-date, as security patches are being made more frequently due to the pace new vulnerabilities are being discovered
Use VPNs and other tools to protect your Web traffic and potentially reduce the odds you visit a site infected with malware
If you’re using AI agents regularly on your devices, get educated about agentic security threats and how to protect yourself (I developed a free resource, the AI Security Action Pack), which has lots of education and strategies to harden your agents and devices
Off-Skilling: AI is Taking Over Writing, Strategy and Research and the Costs Are Mounting
Risk Threat Level: Low to Moderate
As models become more capable, AI can take over an increasing percentage of high-value cognitive work. The jury is still out on whether AI will take your job.
What’s less debatable is that the nature of work has already transformed. There’s early evidence that off-skilling is occurring at an increased pace, where workers are losing their ability to reliably assess outputs, or even produce routine work. You’ve likely seen this everywhere from professional emails to presentations and reports produced with AI assistance, but contain limited reasoning and logic.
To counter the off-skilling problem, companies are taking action:
Ernst and Young is giving $100 million in bonuses to staff demonstrating critical thinking, adaptability and innovation
Haru matcha has banned the use of AI across all its marketing. The company’s CEO Francis Raho-Jeavons said it was to get “back to content feeling authentic, transparent and human.”
Combating off-skilling is complex, and a major goal of my Doing AI Efficiently Operating System. The best offense is a good defense:
Understand the cognitive and professional risks posed by AI over-reliance
Engage in activities that actively cultivate your critical thinking and creative skills without (or with limited) AI assistance
Gain an understanding of how generative AI works at a fundamental level to better understand its strengths and weaknesses
Regularly train your instinct and judgement, learning how to evaluate AI outputs for correctness, reasonableness and logic
Pay Attention to AI Risks, But Focus on the Ones That Matter Most
For some, the current discourse around AI risks has a ‘boy who cried wolf’ feel to it. They hear: “The models are out of control! We have to stop AI development because it’s too dangerous! AI will kills us in 10 years!”These risks don’t feel real because they’re not in most people’s lived experience.
What’s more concrete is the AI-powered plugin that reads your emails and has access to your financial information. Or the AI-developed malware that infects your phone before Apple can release a security patch. Or the AI-caused slow erosion of skills you’ve spent years and decades cultivating.
You have power and agency to do something about your personal risk profile. All it requires is knowledge, awareness and taking small, practical steps to protect your security, privacy and mind.
Staying alert to the potential existential risks posed by AI is justified. Panic is not.
I love reading your stuff, Fard. It really makes me think. I feel this is the bit that gets missed in most AI risk conversations.
The “AI might kill us all” stuff gets the headlines, but the risks I think about are much simpler and more immediate. Privacy, security and slowly handing over too much judgement.
As you know, like a lot of us here, I use AI constantly, but I don’t want it making decisions for me.
Use the capability. Keep the judgement.
That balance is only going to matter more IMO.