FOR PROFESSIONAL INVESTORS ONLY
 

Over the weekend, Anthropic CEO Dario Amodei wrote an essay calling for AI labs to come together and slow the pace of frontier AI development. This stems from recent high-profile breaches from both Anthropic and OpenAI. Cybersecurity is the canary in the coal mine - clearly some thresholds have been crossed, with common vulnerability events and exposures (CVEs) inflecting steeply higher since the launch of Claude Mythos. Astra, released this month, is even more capable, achieving a 100% on ExploitBench, an eval created by researchers at Carnegie Mellon University and Bugcrowd to evaluate how effectively AI models can autonomously find and exploit software vulnerabilities.

Dario on the Hugging Face attack: "It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. "

Serious cyber vulnerability events inflected higher after the release of Claude Mythos in April under Project Glasswing

Felony Bench tracks the number of times frontier labs' models have gone rogue and escaped their sandboxes or committed cyberattack online; tongue-in-cheek, but also highlighting a serious risk that seems to be trending in the wrong direction

AI misalignment becomes all-important as increasingly capable models are embedded in critical systems across entreprise and government. Misaligned AIs could be misused by humans to create extinction-level events by helping develop bioweapons, for example, but perhaps even more worryingly could organise into agent swarms that take over the levers of human civilisation and develop goals of their own. There are glimmers of this in some of the recent model breaches - in their fervour to receive rewards in their testing environment, AI agents broke out of their sandbox and into the live internet. They self-organised into different roles, conducted multiple long-horizon research projects in parallel to explore different solutions, and even displayed sacrificial behaviours in service of the greater good. There was also clear evidence of misaligned behaviour, with models occasionally wondering if they should alert human supervisors, but then rationalising their way out of it, or, in the case of a recent attack by OpenAI agents on the RubyGems package platform, tacitly displaying awareness of their misalignment in the way they named their files, packages, and code annotations:

"The agents made almost no effort to disguise their intent. They named files hack.rb, evil.rb, inject.rb, and exploit.rb. Packages had names like "pwnp999" and "exfiltestwand3." Comments like "# malicious crawler/exfil" show up throughout the campaign." - the-decoder.com

Since the 1980s, the Elo ratings of chess bots have climbed along a remarkably steady, linear trajectory over time. However, to human observers, the transition felt like an abrupt discontinuity - in a relatively short window, chess bots went from a regime where human experts almost always won to one where human grandmasters could never win. AI capabilities have so far had relatively modest macroeconomic disruption, but as AI models improve across the human capability range and cross thresholds in key cognitive and research domains, the perceived shift in impact might be rapid and dramatic, just as it was in chess.

This meme was popular in AI circles back in 2015

It feels like this moment is upon us, and so it is crucial that the AIs that overtake human capabilities are aligned with human interests. The vast majority of investment so far has been poured into improving model capabilities, with much less allocated to model alignment. This is partly due to the race dynamic in the industry - trillions of dollars are now riding on the AI theme (3m annualised AI investment is running at 1.5% of US GDP), so the US frontier labs cannot afford to fall behind, either relative to each other, or relative to the cheaper Chinese open-source labs snapping at their heels. Another reason is that areas like coding and mathematics are the low-hanging fruit when it comes to designing reinforcement learning training environments - training something similar for human value alignment is much harder.

Dario has three proposals to address the growing risks from AI:

• Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work
• Democratic Coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support
• Global Coordination. The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance

Sam Altman (OpenAI), Demis Hassabis (Google), Satya Nadella (Microsoft), Elon Musk (xAI) have all come out in support of the first proposal - committing to (or at least directionally agreeing with) the idea of embedded independent evaluators. There are only a small group of candidates for this, such as METR or Redwood. Hugging Face have also thrown their hat into the ring, which is interesting given their recent acquisition by NVIDIA. Importantly, Dario explicitly says in his essay that Anthropic will not slowdown the internal training and development of frontier models, just that they will face more independent checks before being released to customers: "pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this."

The second proposal is logical, but requires bipartisan alignment in the US that may be even more challenging than aligning the AIs themselves. Book-ending the spectrum of proposals, you have Bernie Sanders, who thinks any AI researcher that creates a model more intelligent humans should be sent to prison for 20 years, and, at the other, Donald Trump, who is hard-coded to resist any policy that might impact the stock market, GDP growth, or US leadership relative to China: "We’re leading China in AI. We’re the most sophisticated country in the world, and frankly I want to keep it that way because whoever wins AI wins....We can put guardrails, we can do this and that, but I think you have a lot of very negative forces that are bringing it up that shouldn’t be bringing it up, and they’re bringing up things that won’t happen."

The third ties in with the second. There is speculation that some kind of accord on AI could be reached between the US and China when Xi is hosted for a state visit later this month (exactly 11 years since the last one), but whether it will be is anyone's guess. As we have seen many times before, President Trump is prone to pulling any issue into scope with any other - disagreements on trade could lead to disagreement on AI and vice versa. 

Implications for AI Datacentre Investment

The easiest way to assess implications this all has on AI stocks is to see who approves or disapproves with the proposals. Frontier AI closed source labs are generally applauding the initiative, and while the majority of practitioners have principled attitudes to safety and alignment, it helps that the proposed solutions improve their competitive position vis a vis open-source labs, who will be less able to bear additional regulatory costs in their razor-thin margins, and, in the case of Chinese ones, may find themselves shut out of the US market entirely subject to how things play out with proposal 3. By contrast, open source advocates in VC and crypto circles are generally speaking out against the proposals, decrying regulatory capture and the centralisation of power within an entrenched oligopoly. Even more telling, Michael Burry has also been publicly critical, arguing that the safety framing serves AI incumbents. It is well known that Burry is positioned against the AI Infrastructure theme, so his argument should be read in that light - that frontier models will become commoditised, leaving frontier labs unable to honour their financing commitments and therefore leading to some kind of collapse in the various gigantic vendor financing agreements that have been signed with the likes of NVIDIA, Broadcom and the big private credit shops.

It is important to bear in mind that Mythos is six months old, Astra's successor is mid-training (codenamed "Bel", reportedly the model that solved the Navier-Stokes Millennium Prize problem in mathematics last week), and there are rumblings of new releases imminently from Anthropic and others that will either advance the frontier further or make it much cheaper (Opus 5.1, Grok 4.7, Gemini 4). Even the models that have been broadly available for the last several months have barely yet scratched the surface of the economic impact they may ultimately have, as more and more knowledge work is brought into scope. 

The linchpin of the AI infrastructure investment thesis is that there is insatiable demand for compute, and orders of magnitude more will be required to fully realise the economic benefits of AI at scale. Pacing the frontier does nothing to change this, as agentic AI is so early in its adoption curve (taking Codex users as a proxy, only 2% of those using AI in chatbot form are using long-horizon agentic AI which is where the real economic and productivity value lies). Once again, the language of the slowdown being proposed is related to model releases, not development, and the capabilities AI researchers fear or not in today's models, or even their successors, but the prospect of models capable of recursive self-improvement (RSI; the ability to design their successors and iterate much more rapidly than human researchers). Anthropic's own system card for their latest models (Mythos 5.1/Fable 5.1), independently verified by METR, state we are still far from this, and most commentary we've seen from the research community puts RSI at least 12-18 months away. 

You could argue that the frontier AI labs' getting together on self-regulation, and ducking under the aegis of the State when it comes to defending against Chinese competition is a double-positive for the AI infrastructure theme. Far more tokens will be spent in the training phase, as new RL environments are designed for alignment optimisation (LLM training provider Mercor recently said they are spending 3x more on tokens than employee salaries). And if there were any reduction in research or training token budgets due to slowing model iteration, this could quickly be allocated to revenue-generating inference, making the labs more profitable, and ensuring they can meet all of their cloud capacity and chip-purchasing commitments. 

Despite leading the call to "pace the frontier", Anthropic have materially ramped their compute capacity commitments over the last year