Syntax Pass Rates Climbed From 50% to 95%. Security Pass Rates Never Moved.

My first job was a checkout lane at Kmart when I was in high school. It paid for gas and Taco Bell, and by the numbers I was extremely good at it.

Corporate sent down a directive that cashiers needed to move customers through faster, and they were specific about the measurement: elapsed time from the first item scanned to the moment you marked the scanning done and read the customer their total. So that's what we improved. We'd unload the entire cart first, line everything up on the belt with the barcodes turned to where our hands would be, and only then start scanning. Fast. The stopwatch loved us.

The customers waited longer. They stood there watching a teenager arrange their groceries for the benefit of a clock that hadn't started yet, and the interval that actually mattered to them – standing in a store, wanting to leave – got worse. Corporate was thrilled with us. We won awards.

Nobody was cheating. That's the part that usually gets lost. Every one of us was doing exactly what we'd been asked to do, as well as we could do it, and the work didn't disappear when the number improved. It relocated to the part of the process nobody was counting.

I kept thinking about that belt while I read Veracode's testing on AI-generated code, because the same mechanism is sitting in the middle of the data.

Two curves, three years, and one of them didn't move

Veracode runs a fairly simple test, in versions going back to 2023: hand a model a coding task that has a known opening for a vulnerability, then check two separate things. Does the code compile? Is it secure?

The first question has stopped being interesting. In Veracode's 2026 GenAI Code Security Report, published at the end of July, models generate compilable code at what Veracode's own write-up calls a near-universal syntax pass rate of about 100%. The second question is where the report lands: across four testing snapshots and more than 100 models tracked since the program began, the average security pass rate sits at 56%, virtually unchanged since last year's report. With no security-specific prompting, models fail nearly 44% of the time.

The individual results don't rescue it. GPT-5.5 leads at 68%, six of the eleven models in that round cluster between 50% and 53%, and Qwen3.7-max comes last at 50%. By Veracode's own summary, the best model available today still fails nearly one in three security tasks.

The trend line lives in the earlier Spring 2026 update from March, and that's the piece I can't stop looking at. Syntax pass rates climbed steadily from about 50% to 95% over three years. Over exactly the same stretch, security pass rates stayed essentially flat, hovering between 45% and 55% regardless of model generation or release date. Those are two different test rounds, so keep the figures apart – 95% syntax is March's reading and roughly 100% is July's, which makes the 50-to-95 climb the three-year trajectory rather than today's number.

The March round covered 80 coding tasks across four programming languages and four critical vulnerability types, with security flat across more than 150 evaluated models, though I'm taking those design details from NHIMG's summary rather than Veracode's primary, so treat the model count as approximate.

One curve went from coin flip to solved. The other one is still a coin flip.

Why one curve moved and the other stayed put

Compilation has a free, instant, unarguable judge. Every time a model produces code, a compiler or interpreter can say within milliseconds whether it's valid, and that verdict needs no human, no surrounding context, and no debate. A signal like that can be generated at effectively unlimited volume, fed back into training, and used to rank one output against another. Three years of that pressure gets you to 100%.

Security has no such judge. Whether a line of code is safe depends on what the code is for, where its data came from, who else can reach it, and what somebody hostile would try. Static analysis catches a slice of that, which is genuinely useful and also a slice. The rest requires a model of purpose and a model of an adversary, and neither of those is available to grade a training run at scale.

Which puts me back on the belt at Kmart. The stopwatch measured the scanning interval, and the scanning interval got much better, because that was the property the measurement could see. The unloading didn't get better. It got bigger, because it was the place work could go without showing up anywhere. Improvement flowed to the measured property and skipped the unmeasured one, and nothing about the diligence or good faith of the people involved changed that.

The models did the same thing. Capability improved sharply on the property a machine can evaluate automatically, and stayed still on the property that requires knowing what the code is meant to do. Nobody involved was careless. The mechanism was the measurement.

I want to be careful about how far to push that. This is one domain, one vendor's harness, and a task set somebody chose, so it's a well-supported reading of code generation rather than a proven rule about machine capability generally. As a lens, though, it earns its keep. When you're guessing which parts of a workflow AI will get good at without anyone deciding to make it good, ask what can grade it for free. Properties with a cheap automatic grader tend to improve on their own. Properties that need somebody to know what the thing is for tend to sit exactly where they were.

Neither scale nor specialization moves the number

The comfortable hypothesis is that this is a maturity problem and better models will fix it. The data says no, in two directions at once.

Coding-specialized models average a 51% security pass rate. General-purpose models average 52%. Being trained to write code faster, as Veracode puts it, does not mean writing it safer. Size doesn't help either: per the release accompanying the report, large models over 100 billion parameters average 53%, medium models 51%, and small models 51%. Those gaps are small enough to ignore.

One partial exception is worth naming, because it fits the pattern rather than breaking it. Most standard models clustered near a 55% security pass rate while some reasoning models reached 70% to 72% – a figure I found in Larridin's summary of the testing rather than in Veracode's own material, so hold it loosely. Models that spend inference time deliberating do better at the property that requires reasoning about intent. That's the shape you'd predict if the lens above is right.

Models are strongest where the training material is thickest

The sharpest thing in the July report is a breakdown that almost nobody covering it has written about. Security pass rates aren't uniformly mediocre. They vary enormously by vulnerability class: 83% on SQL injection and 87% on cryptographic algorithms, against 15% on cross-site scripting and 12% on log injection.

I have never personally found a cross-site scripting bug in my life. But I can read that spread. SQL injection has been the worked example in every security tutorial, every code review comment, and every conference talk for as long as most of us have been working, and the fix is short and canonical. Parameterize the query. Choosing a cipher is similar: there's a right answer, it's the same right answer nearly everywhere, and the corpus contains it thousands of times over, labeled as the right answer.

Cross-site scripting and log injection don't work that way. Whether a string is dangerous depends on where it lands – which template, which sink, which encoding context, whether some other layer already escaped it. Two lines that look identical differ in safety because of facts that appear in neither line. That's the intent problem again, now compressed into a single statement instead of spread across a system.

The language spread has the same texture. Pass rates run from Python at 63% down to Java at only 30%, last by a wide margin, though Java shows the clearest improvement trend of the languages tested.

The vendor has an interest, and the finding survives it

Veracode sells application security testing, and its report concludes that you need security review of AI-generated code. The recommendations section is a product argument.

The divergence survives that, which is why I'm willing to build on it. The number that would be easiest to shade is the absolute 56%, since it depends on task selection and grading criteria a vendor controls. What I'm relying on instead is the relative movement between two properties, measured by the same harness on the same models over the same period. To bend that toward a security vendor's interest you'd have to suppress security improvement that was really happening while letting syntax improvement through, across four snapshots and more than a hundred models. The comparison is much harder to tilt than the headline is.

Where to put the human attention you have

If your quality gates are compilation, linting, and test-pass rate, they are watching the curve that fixed itself. A gate that's gotten greener every quarter for three years may be reporting the health of the property that improved on its own, and telling you nothing at all about the property that didn't.

The practical read isn't "add a security scan," though by all means add a security scan. It's that human review time is a fixed budget and belongs on the kinds of correctness no automatic grader covers. A reviewer reading for syntax and style is duplicating a machine that's better at it than they are. A reviewer reading for what could go wrong given what this code is for is doing the only job nothing else in the pipeline does. That's a hard reallocation to ask for when the review that isn't happening is already common and the verification gap keeps widening, and it's the same place I end up coming from every other direction too: capability isn't the constraint.

Most of what I do at Fieldway is Research and Strategy rather than security review, but the question underneath is identical. What does the evidence actually cover, and what is it quietly leaving to somebody's judgment?

I got an award for making customers wait longer. The award was real and the measurement was real and the improvement was real too. It just belonged to a much smaller thing than anyone thought they were measuring.

If you want to talk through where that line sits in your own delivery process, email matthew@fieldway.org.

Sources

• Veracode, Spring 2026 GenAI Code Security Update
• Veracode, 2026 GenAI Code Security Report
• Veracode, 2026 GenAI Code Security Report: AI Is Writing More Code
• BusinessWire, LLMs Are Getting Smarter, But Not Safer
• NHIMG, GenAI code security is stuck while syntax quality keeps improving
• Larridin, How to measure AI-generated code quality