Public fear often centers on Artificial Intelligence going rogue, inspired by sci-fi visions of robots taking over. However, a more immediate risk is misuse by people deploying AI for unintended purposes. Recently, Anthropic—the company behind the AI model Claude—reached a critical decision. CEO Dario Amodei has openly declined new contract terms from the US Defence Department. His rationale: the Pentagon seeks unrestricted access to Claude, potentially for generalized surveillance or even automated weapons, contrary to Anthropic’s ethical constraints.
The Power of Refusal
Understanding this moment requires recognizing the pressures AI startups face. These companies operate under intense financial demands. Defense contracts, which can be worth billions and guarantee long-term security, are highly sought-after. Anthropic’s refusal to accept such a deal—citing risks in the terms offered—is significant. The company sees accepting unrestricted use as potentially harmful, akin to a car company refusing to equip vehicles with weapons, regardless of the potential profits.
Amodei’s objection centers on ‘unfettered use.’ Most software products come with an End User License Agreement (EULA) that outlines permitted and prohibited activities. Anthropic’s EULA specifically prohibits high-risk uses, such as surveillance via facial recognition or control of weapons systems. The Pentagon’s new terms would bypass these safeguards, essentially asking for a version of Claude stripped of ethical restrictions.
The Specific Dangers
The concern over the military’s adoption of AI centers on the scale and speed at which these systems operate. AI models like Claude process information exponentially faster than humans. In a surveillance context, such a system could theoretically monitor all communications in real time, identifying potential dissent throughout a population. This immense power threatens to shift the balance between individual freedoms and authoritarian control.
The issue of autonomous weapons is particularly troubling for AI safety experts. Allowing AI to control weaponized systems, such as drone swarms, with open-ended instructions is risky—especially given the technology’s tendency to make errors. Anthropic argues that current AI is not reliable or predictable enough to be entrusted with lethal decision-making, regardless of who seeks to use it.
A Split in the Industry
This incident highlights a deepening divide within the tech industry. Some companies aim to integrate their technologies into national defense and actively seek military contracts. Others, like Anthropic, prioritize safety and view their innovations as powerful tools requiring strict oversight to prevent misuse and societal harm.
By resisting these terms, Anthropic hopes to set an industry precedent based on ethical conduct and safety. The company is taking a risk, wagering that its reputation is more valuable than the immediate rewards offered by a defense contract. If others take the deal, Anthropic may lose business, but it maintains its principles. For now, Claude remains a civilian tool—serving people, not power systems.
