We frequently hear from pharma/biotech leaders that AI has the potential to radically accelerate the pace of drug discovery, and, maybe, eventually cure all disease.

If you know anything about biochemistry, you’d quickly realize that judging by the current behavior of pharma/biotech, no one really believes that.

Because if they did, they’d be obsessed with the questions of bio and chemical safety of these AI models just as much as Anthropic is obsessed with the general problem of alignment.which might not necessarily solve the problem of bio/chem safety because, for example, we’re not using LLMs to fold/co-fold proteins

We got to the point of me writing this fugue not by OpenAI or Anthropic explicitly trying to train an LLM capable of sophisticated cyber attacks. No, the surprisingand maybe not so much in hindsight aspect was that frontier labs simply tried to make models better at coding, and as a byproduct, they became world class at finding vulnerabilities in software. Models got so good that Anthropic, by their own account, decided to institute a limited preview to trusted partners through Project Glasswing instead of releasing the model to the general public.

Some of you might already see the parallel.

You see, curing a disease often requires finding a good molecule that would bind to a protein that misbehaves and neutralize it. And although we’ve made some strides in predicting whether a molecule can bind to a protein, there’s still a lot of work to be done to ensure the molecule is easy to make, does not have off-target effects (does not neutralize the proteins it’s not supposed to neutralize), and has a wide enough therapeutic window. But ultimately, if you claim to believe that AI will be able to cure all disease, you’re effectively stating that we’ll have a system that, given a specification of a protein of interest, creates a molecule that satisfies all those constraints.

Creating a perfect chemical weapon is just applying that same system not to a misbehaved disease-causing protein, but to a regular, well-behaved, likely life-critical protein.

In other words, it is almost a given that just like with the Anthropic Mythos model getting extremely capable at cyber attacks after getting better at writing code, IF/WHEN we have a sufficiently powerful AI model capable of end-to-end drug design, the same model will be extremely capable at creating chemical weapons.

And sure, just designing a molecular structure is probably not enough, you need to make it. And so maybe, the defense here is that just like with nukes, the difficulty is not in knowing how to make a nukewhich has been proven with say Nth country experiment but in the iterated engineering required to get the system just right and access to the right materials. But at least this should be an explicitly formulated position in an ongoing debate. When was the last time you heard anyone seriously talking about how they’d prevent their powerful AI Drug Design platform from being used for chemical and biological weaponry? Benchmarking an LLM if it refuses to respond to chemical weapon prompts does not count as serious preparation.which is often done by simply classifying prompts before they’re passed to an LLM and often results in refusal to answer even regular chemical prompts You’re not going to create a chemical weapon by prompting an LLM, no matter how clever it is, the problem lies beyond textual representation and requires working with structures of proteins and molecules as first-class citizens. If this problem were taken seriously, I’m not sure AlphaFold3/Boltz/Chai would be open sourced. For now, they’re more focused on proving these models can work with “good targets”. The problem, of course, is that once we have tech that works for good targets it’ll be too late to worry about alignment.

On a broader, more utilitarian point, with the dooming AI-induced cyber security crisis, the society at large might be tempted to question how much do we really need these fancy code-writing models if they create so many issues.I’ve argued, people catastrophically underestimate the benefits that abundant software can bring And maybe our approach to solving this crisis should be instructed by a realization that we, most likelyI personally believe it’s a matter of WHEN, not IF, will have to deal with the same question in AI-driven drug discovery, only then no one would have any doubts about the positive case (curing disease is what everyone understands to be a good thing).

But are you really willing to cure all disease at the cost of creating the ultimate chemical weapons designer?