According to a report by TechCrunch by Lorenzo Franceschi-Bicchierai, the guardrails and strict vetting programs established by AI giants to prevent malicious hackers from exploiting their models are beginning to complicate the work of legitimate network defenders and offensive cybersecurity researchers. These researchers, whose role is to identify vulnerabilities in systems and develop ways to exploit them before criminal actors do, are running into blocks from vetting mechanisms or facing inconsistency in system responses. These restrictions often lead to the opposite result, forcing researchers to abandon regulated American systems and transition to using open-source models, sometimes from foreign companies, which do not include any vetting.
Export Controls and Vetting Programs of Tech Giants
Over the past several months, AI giants have developed special vetted programs and strict guardrails to limit the use of their models by malicious hackers. In June, the U.S. government imposed export control restrictions on Anthropic's much-hyped AI models, known as Mythos and Fable. This step was prompted, at least in part, by a report claiming that it was possible to bypass the guardrails of these models, which were designed to prevent users from using them to build and execute malicious cyberattacks.
Regardless of whether the incident was indeed motivated by fears of a jailbreak, the fact is that Anthropic has repeatedly marketed the Mythos model as some kind of "doomsday cybermachine" that can only be given to carefully vetted users, and even then with strict guardrails implemented in practice. (The export controls on Fable 5 and Mythos 5 have since been lifted; Fable 5 returned to general access on July 1, while Mythos 5 has been reintroduced only to vetted U.S. organizations as part of the government's review process.)
This type of gatekeeping is not unique to the Mythos model. Both Anthropic, with its other models, and OpenAI offer cybersecurity researchers special programs they can apply to in order to get vetted and—if approved—receive access to models with fewer cybersecurity restrictions. OpenAI's program is called "Trusted Access for Cyber", while Anthropic's program is named the "Cyber Verification Program".
Criticism from Offensive Security Researchers
These restrictions have drawn widespread criticism, particularly from researchers whose job is to find unknown security vulnerabilities in systems and find ways to exploit them before cybercriminals do. During a recent appearance on a cybersecurity podcast, Mark Dowd, a well-known security researcher, said: "it's not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what's not." Dowd has spent decades finding and selling "zero days"—previously unknown software flaws and the exploits that take advantage of them—to Western governments, rather than reporting them to software makers so they get patched. Governments pay premium prices for these vulnerabilities precisely because they stay open, which aids intelligence operations. Dowd admitted that the nature of his work might make him biased, but he is not the only one holding this view.
"Like a Hammer": A Tool That Is Both Offensive and Defensive
Offensive cybersecurity researchers—those who proactively probe systems to locate weaknesses—described to TechCrunch how they use AI tools and deal with their guardrails. Chris Anley, the chief scientist at security consulting giant NCC Group, noted that asking an AI model to try to exploit a bug is a key step in confirming that a real vulnerability worth fixing actually exists. But if a guardrail causes the model to refuse to answer the question outright, the restriction hurts defenders.
"This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base," Anley explained. "So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can't really be unpicked." Anley compared the tool to a hammer: "You can't build a house without a hammer. It's definitely a tool but it's also irreducibly a weapon as well." When he and his colleagues run into such a roadblock, they sometimes turn to open-source AI models that come with no guardrails at all.
Concerns Over Data Leaks and Moving to Local Models
Paolo Stagno, the chief technology officer (CTO) of CrowdFense, a well-known company that develops, acquires, and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying that AI companies "essentially treat customers like children who need babysitting" with their vetted programs and guardrails. Stagno noted that he and his colleagues use frontier models—but solely for reverse engineering. They avoid using AI to help find vulnerabilities or build exploits because feeding these materials into a cloud-based model risks leaking sensitive vulnerability data or having it absorbed into future model training runs. For this stage, he said, they use open-source models run locally, as they do not rely on sharing data outside of the model.
Another security researcher, Giuseppe Cali, who finds zero-days and develops exploits, noted that guardrails do not impede his work. This is because he does not use AI for offensive work; instead, he uses it for initial reverse engineering, to understand the code he is analyzing, and to build supporting tools. For these tasks, he explains, AI tools can speed up the process and allow him to focus on discovering vulnerabilities. “I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow," Cali said. "I am jealous of my bugs, and I like this game too much to let models play it for me.”
System Inconsistency and the Push Toward Chinese Models
One researcher at a smartphone-component manufacturer, who spoke on condition of anonymity because he is not authorized to talk to the press, said his employer is not part of Anthropic's CVP program, and as a result, its tools are barely useful for finding vulnerabilities because the guardrails are too strict. "If it catches wind we're doing anything security related, it just stops and isn't usable," the person said.
Chris Thompson, chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con—an offensive security and AI-focused event—shared that in his experience using frontier AI models, the guardrails can be inconsistent and work differently every day. This is true even inside the looser boundaries of Anthropic and OpenAI’s vetted programs.
"I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program," Thompson said. "Instead of analyzing a vulnerability and reasoning through the exploitability, you're trying to find why you're getting inconsistent results or why are models over-sanitizing the output."
As a consequence, researchers rely on or are pushed toward Chinese open-source models like GLM—freely downloadable models that can be run locally with no vetting or usage restrictions, Thompson said. "You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems," he noted. "I think it's more harmful than good to have these guardrails in place."
Rather than tightening restrictions further, Thompson called for the leading AI labs to open up their programs, provide responsible access, and also hold those who abuse their tools accountable. Otherwise, he argues, defenders will lose the AI race. "There's this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before," Thompson said. "But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now."