SecurityBrief UK - Technology news for CISOs & cybersecurity decision-makers
United Kingdom
AI coding benchmarks show cheaper models can be safer

AI coding benchmarks show cheaper models can be safer

Tue, 17th Nov 2026 (Today)
Joseph Gabriel Lagonsin
JOSEPH GABRIEL LAGONSIN News Editor

Secure Code Warrior has released new findings from its SCW AI Trust Index on the security of AI-generated code, covering tests of more than 30 AI models.

The research found that curated security rule sets reduced measured risk in AI-generated code by as much as 70%, while vague prompts produced less reliable results. It also found that lower-cost models sometimes generated less risky code than flagship systems, challenging assumptions that higher prices or newer models produce safer output.

Secure Code Warrior updated its testing approach to assess models inside a harness designed to better reflect real software development use. It also added comparisons of functional completeness alongside security results to show both the risk and utility of AI-generated code across software development tasks.

In total, the study used AI systems to build more than 15,000 applications and 41.3 million lines of code across 11 programming languages. It examined models from OpenAI, Anthropic, Google, Alibaba, Mistral and others.

Across the models tested, output ranged from 0.24 to 3.66 vulnerabilities per 1,000 lines of code, a 15-fold gap between the best and worst performers in the benchmark.

The findings come as software teams rely more heavily on large language models for coding work. Secure Code Warrior cited external survey data indicating AI is expected to account for 65% of committed code by 2027, increasing pressure on security teams to understand the risks tied to different models and prompting strategies.

Prompt effects

The study found that the prompts and rule sets given to a model had a material effect on the security of the code it produced. According to the results, stronger models benefited more from curated rules than weaker ones.

Among the lower-risk half of models, Secure Code Warrior's own rules and third-party rules reduced measured risk by an average of 54.15%. For the higher-risk half, the average reduction was 34.05%.

The benchmark also found that no single model led across all frameworks tested. Fifteen different models ranked first, or tied for first, in at least one of the 11 frameworks at baseline, while GPT-6-Astra-High led or tied in four.

Cost and risk

One of the more striking comparisons in the data involved the relationship between price and security risk. Some budget models carried lower measured risk than premium models even without additional rules.

In one example, GPT 6 Luna recorded a mean risk score of 19.4 at USD $0.10 a run, compared with a mean risk score of 43.6 for Claude Opus 5.5 at USD $8.17 a run, using the models alone without rules.

The result suggests procurement decisions based only on model branding, headline performance or price may not reflect actual security outcomes in coding tasks. It also underlines the challenge for companies trying to standardise AI coding tools across large engineering teams.

Separate academic work has pointed to a lower vulnerability rate for human-only commits, although the methodologies differ. Secure Code Warrior referenced Stanford University research that found human-only commits introduced roughly 0.08 vulnerabilities per 1,000 lines, compared with the range seen in the AI benchmark.

Pieter Danhieux, Co-Founder and Chief Executive Officer at Secure Code Warrior, said the results showed that assumptions about newer or more expensive models do not always hold up.

"It's easy to assume every new AI model writes safer code than the last, or a frontier model is more secure than a budget model. Our updated findings show this isn't always the case," said Danhieux, Co-Founder and Chief Executive Officer, Secure Code Warrior.

He said the company's latest research highlights the importance of governance around how AI coding models are used.

"Today's models know approaches to secure coding, but they don't do it by default. The harness, prompts and rules they're given decide how secure the output is, and that behaviour can shift with every new release. A one-time benchmark can't keep pace, which is why the SCW AI Trust Index evolves alongside the models themselves. Security leaders and developers need a current, evidence-based view of the risks they're inheriting that is verifiable by the languages and frameworks in use within the development team," said Danhieux.