Every time I read that Europe will “never catch up” in AI, it annoys me. The distance is too large, the gap is permanent, we should just accept American dominance and move on. I keep hearing it from commentators, from tech media, from people who should know better.

It is nonsense.

We have a large population of highly educated, intelligent people. We have a culture that, for all its flaws, raises people to think about ethics, about climate, about responsibility toward fellow humans. Yes, we need to live up to those values more than we do. But we also massively underestimate ourselves. There is this persistent European reflex that everything is possible in America and that you have to move there to “make it” (whatever that means: getting extremely rich while ignoring the people sleeping in the street meters from your car, probably). If you think about it, that is not what most Europeans want. Too selfish, too wasteful, too careless about people and planet.

The premise that we are hopelessly behind is not supported by anything except the marketing of the companies that happen to be leading right now.

I Have Switched Models Several Times

If we go back a long way (it feels like that, but it is only about 3,5 years) I started with GPT-3.5 for coding when it launched in late 2022. After that GPT-4. Then 5, then Opus. Every time I switched, the new model was better than the last, but never as good as marketed. Every time, the model I left behind had been called “the one that will never be surpassed” at launch. And every time it was said that developers would be out of a job in the next 6 months. None of this was true.

The Artificial Analysis leaderboard (updated regularly, and any snapshot is already outdated by the time you read this) tells a consistent story: the distance between American proprietary models and open-source alternatives is much smaller than the marketing suggests. Models that launched a year ago are routinely surpassed by competitors that barely existed at the time.

Why Does the “Hopeless Gap” Story Persist?

So why does the “hopeless gap” story persist? Because it serves everyone who tells it.

OpenAI and Anthropic both filed S-1s (the registration to become publicly traded on the stock market) within a week of each other in early June 2026. OpenAI is targeting a valuation of up to a trillion dollars. When you are about to go public at those numbers, being mysterious about your capabilities and maintaining the narrative that nobody can touch you is not just marketing, it is a financial necessity. “We are so far ahead nobody can catch us” is the pitch, and every headline about export restrictions and “too dangerous” models reinforces it.

The export controls are a neat arrangement where everybody wins. Politicians get to look patriotic, protecting American interests and national security (and probably making some money along the way, given this administration’s track record with conflicts of interest). Companies get free headlines about how powerful their model must be if the government is scared of it, which is great for your stock price when you are about to go public. The cycle is predictable: a new model launches, it gets restricted as “too dangerous,” and then restrictions quietly loosen just in time for the next model to take its turn. Media repeats the narrative because “permanent American dominance” is a better headline than “incremental progress across multiple geographies.”

And the anxiety it creates in Europe is real: we believe the marketing from those huge American companies and ‘slikken het als zoete koek’ (swallow it like candy, as we say in Dutch). The gap is there, but the perceived width of the gap is not real, and it is becoming a self-fulfilling prophecy.

What the AI Timeline Actually Shows

If you zoom out even slightly, the pattern is obvious. This field is very, very young: about four years old in any practical sense. In those four years:

• Google went from “their AI is embarrassing” to a serious contender in under two years
• Anthropic went from “ChatGPT knockoff” to having models restricted by the US government in three years
• DeepSeek went from unknown to shaking markets in about a year
• Mistral went from a French startup to a model provider aiming to compete with the big three

None of this is consistent with the idea of a permanent, uncatchable lead. It is consistent with a young, fast-moving field where leadership rotates and spending advantages erode quickly as the technology commoditises.

I wrote earlier today about European cloud providers running open-source models as a practical alternative to US hyperscalers. That post was about the solution. This one is about the fear that makes people think they do not need one: the idea that the race is already over.

It is not. The race barely started, and the starting positions change every year. If one to two years of lag were truly fatal, most of the models we use today would never have existed. And more exciting: what is possible in a year? How do we create and use these models in an ethical and compassionate way?

Working on AI strategy and wondering how the competitive field affects your choices? Get in touch to compare notes.

Sources

• Artificial Analysis LLM Leaderboard – Independent model comparison, updated regularly
• Fortune: Anthropic disables Fable/Mythos – Export control whiplash
• CNBC: Mythos 5 restrictions loosened – Two weeks later
• AP News: AI data centres and water consumption – The resource cost of the spending race

Related Posts

• The Third Option for Enterprise AI – The practical alternative to US hyperscalers
• Why LLMs Will Not Have Your Next Big Idea – More on separating hype from capability

Almost all AI services I use with the different companies and governments I worked with are American based, mostly Azure and Anthropic. As we are all aware this is not a situation that is acceptable: especially not with ‘the orange child in the white house’ spewing nonsense every day and keeping us on our toes, American companies are no longer a trustworthy partner. They obviously chose sides during the orange child’s election, marking a departure from the heralded status American tech companies had over the last decades.

So all kinds of solutions are proposed to solve this. And basically there are two main trains of thought (or perhaps three). Option one: keep using the big American cloud providers (OpenAI, Anthropic, Google) and accept that your data crosses the Atlantic. As you can derive from the previous paragraph this is technically an option, but in practice definitely not: trust has been broken and will take many years to rebuild (if at all, given the current direction of American foreign politics). The alternative that I get to see pitched a lot: go fully local, run your own hardware, keep the data in the building. (Could be an option, I was in that camp for a long time, but what I do not understand about this proposition is that those who propagate it, do not mention that this would only be logical if we would move back from the cloud to full on-prem hosting of all data. For various reasons I do not see this happening anytime soon)

So that leaves a third option that makes more practical sense to me and to most European companies and is a big business opportunity within Europe: hosting opensource LLMs on European servers by European companies, that make sure privacy and data governance are guaranteed.

Option 1: Stay on US Hyperscalers

The current situation is using US companies for LLM: it works, it is what we are used to. However this is not sustainable, both in costs and in privacy and risking being cutoff of the service on a whim of the orange child.

Given that it is hard to filter PII or medical or legal data from a prompt, it is a minefield of GDPR and privacy issues that you need to navigate. And that is assuming you are able to make sure your employees only use your approved methods (do not forget your employees are infinitely creative to work around them, so you better make the approved methods really easy to use and perform really well, otherwise you could as well not have bothered at all)

Option 2: Go Fully On-Prem

The counterargument is to bring everything in-house. Buy GPUs, rack them in your server room, run open-source models, keep all data on your own hardware. Full control, full sovereignty, zero legal risk.

The edge AI market is real and growing, and there are genuine use cases for local inference: air-gapped environments, defence, certain healthcare scenarios.

But here is where it gets illogical for most companies. These same organisations run their email on Microsoft 365, their CRM on Salesforce, their databases on Azure or AWS, their file storage on SharePoint. They trust cloud providers with all of that. And then the argument is that specifically the LLM part, the thing that processes text and returns text, needs to be on physical hardware in the basement?

If your data governance concern is serious enough to run AI on-prem, it is serious enough to move everything back on-prem. And that is probably not going to happen.

There is also the cost problem. Running inference locally requires expensive GPUs, it is really complicated, so you need people who know how to operate them, and continuous investment to keep up with model improvements. For companies that are not in the AI infrastructure business, this is a distraction. (even though personally I believe every company should be in full control of its own data and software and I know this belief is shared by many IT and data professionals, this is unfortunately not a shared vision across most boards)

Option 3: European Providers, Open-Source Models

The option I find most interesting is getting discussed more and more, but not yet seriously enough: European cloud providers running open-source models under European jurisdiction.

The landscape here has changed dramatically in the last two years. Mistral AI now offers models under Apache 2.0 with inference running entirely in EU data centres. Scaleway offers managed GPU instances with model-as-a-service APIs under French legal jurisdiction. Deutsche Telekom invested over a billion euros in an Industrial AI Cloud with 10,000 GPUs in Munich. IONOS offers dedicated GPU servers. Even smaller players like Regolo.ai provide GDPR-compliant inference with zero data retention policies.

And the models are good enough. Open source models compete with proprietary models on most practical tasks. For the “boring” enterprise use cases that make up 90% of actual AI adoption (document processing, summarisation, classification, translation, code assistance), open-source models hosted in Europe would do the job.

And costs of inference are dropping rapidly: Epoch AI’s research shows inference costs dropped 1,000x between 2021 and 2026 for GPT-3 level performance. The median decline is 50x per year, accelerating to 200x per year post-2024. GPU compute cost follows Wright’s Law with an 89% learning rate per doubling of cumulative production. That is faster than solar panels ever dropped (around 20% learning rate).

Why This Makes Sense

The logic is simple. If you already trust cloud providers to run your email, your databases, your file storage, and your business applications, then the question is not “cloud versus on-prem.” The question is “which cloud provider, under which legal jurisdiction.”

A European provider running opensource models under EU law gives you:

• No Chapter V transfer headaches
• No CLOUD Act exposure
• Full EU AI Act compliance (high-risk obligations enforceable from August 2026, with extended deadlines into 2027-2028)
• The same operational model your IT team already knows (APIs, managed services, SLAs)

The market agrees: Gartner reports that 61% of Western European CIOs plan to increase reliance on local cloud providers, and European sovereign cloud spending is projected to grow 83% year-over-year to $12.6 billion in 2026. The Cohere acquisition of Aleph Alpha, backed by $600 million from Schwarz Group, signals that serious money is flowing into this space.

When This Does Not Work

Frontier reasoning tasks. If you need the absolute best model for complex multi-step reasoning, agentic coding, or advanced multimodal work, US providers still lead. The gap is closing, but it is definitely still there. As a rule of thumb: the best opensource models are about a year behind the best closed source models. Last January I noticed Opus was at such a level that I could actually start using it as I hoped, so it is reasonable to assume that coming January I will be able to do the same with an opensource model: exciting! An interesting complication is the recent US government blocking of Fable 5 and GPT-5.6. If the American government keeps pulling its own models from the market, the “first mover” advantage disappears, and investment in open-source alternatives accelerates. But that is a topic for another post. Given the year advantage, we will be having that same functionality in an opensource model in June 2027 anyway.

For most enterprise workloads though, “the best model” is not the relevant criterion. “Good enough, legally clean, and operationally simple” is. And that is where European providers are becoming competitive.

The future I see is not a binary choice between American clouds and on-prem hardware. It is trustworthy European partners that run the LLM part, just as well as the other cloud services a company needs. And as a benefit to keep going that path: the European rules are among the toughest in the world, if a company can meet them here, it is quite possible to offer these services in other parts of the world, like South America and Africa.

Thinking about where to run your enterprise AI? Get in touch to talk through the options.

Sources

• Epoch AI: LLM inference price trends – 1,000x cost reduction data
• Wright’s Law extended to GPU compute – 89% learning rate vs solar’s 20%
• EU-US Data Privacy Framework – Current legal basis for transatlantic transfers
• GDPR and EU-US data transfers guide 2026 – Post-Schrems II compliance landscape
• EU AI Act – Full enforcement August 2026
• Gartner: Europe accelerates sovereign AI shift – 61% of CIOs increasing local provider use
• Edge AI market projections – $25B (2025) to $143B (2034)
• Cohere acquires Aleph Alpha – $20B combined valuation, sovereign AI focus
• Deutsche Telekom Industrial AI Cloud – EUR1B+, 10,000 GPUs
• OpenAI and Anthropic limit new AI models to Trump-approved customers – Washington Post on US export controls
• VS blokkeert Anthropic Claude Fable 5 en Mythos 5 – Tweakers (Dutch) on the same topic

I have been using LLMs daily for years now, mostly to code, but also for research, writing, and analysis. They are genuinely useful, sometimes spectacularly so. I have built tools, workflows, and entire coding pipelines around them. So this is not a “LLMs are bad” post. This is about something I have been thinking about for a while, and an enthusiastic youtube video about the LLM Wiki pattern got me thinking again about the drawbacks (besides the obvious benefits) and why I think that way of gathering and using data is not good enough for me.

In this post I will take you through two ideas: the first one is ‘LLMs can not create truly new ideas’ and the second is that ‘the simplification of information in an early stage leads to suboptimal results’. And the end conclusion is that, although we can do magnificent things with LLMs, this technique is not able to solve any real problems and will always result in mediocrity.

The Argument From Mechanism

Next-token prediction works by sampling from a distribution fitted over the training corpus. There is no reasoning faculty that could exceed what the text encodes. The model stores a very high-dimensional interpolation function over token sequences. It produces outputs for inputs not exactly in the training set (that is what makes it useful), but it interpolates within the convex hull of expressed human thought. It does not extrapolate beyond the manifold it was fitted to, and the model does not learn.

What I here define as a good idea is a genuine breakthrough that is good because it violates the regularities the corpus taught. That is structurally outside the region the model can reach. To the model, the revolutionary idea and the incoherent error look the same: both are low-probability under the learned distribution. There is no mechanism to favour “low-probability because true-and-new” over “low-probability because wrong.”

The model’s only notion of “good” is “high-probability under the learned distribution,” which means “resembles what was already thought.” That is anti-correlated with genuine novelty almost by definition. The quality filter is filtering for conventionality, which explains why every LLM output feels competent but unsurprising. Ask any LLM to build you a web app and you will get React, Tailwind, and shadcn/ui. And only if you are an expert (or have a good general knowledge) you will spot the things it misses or simply fails on (your knowledge on that domain is then above medium).

The Argument From Scale

Now you could argue that there are a lot of good solutions in a LLM (including the novel ones we need to solve important issues like climate, energy and water management). And that they are simply buried in the vast knowledge of the LLM and need to be combined over different domains and then extracted. But if recombinational novelty could produce breakthroughs, the current reality would have surfaced them. We have the largest idea-discrimination experiment ever running right now, with millions of LLM instances running daily, with humans reading and judging. If LLMs would live up to their promise (the big breakthrough for humanity speech we have been hearing for a few years now): why do we not see those novel ideas showing up?

The result of those LLMs has a different character: a flood of interpolative novelty (restatements, syntheses, transfers of known methods to adjacent problems) and nothing on extrapolative novelty (fundamental breaks, genuinely new ideas). That bimodal signature is exactly what the mechanism predicts. Abundant where the manifold is dense (between known points), absent where it is not (beyond them).

If the good recombinations were reachable but merely unrecognised, millions of human discriminators sifting the output over years would have found them. They have not. The breakthroughs are not hidden in the output. They are simply not there.

Recent research backs this up. Chakrabarty et al. (2025) found that LLM-generated ideas lead to homogenous outcomes across domains, and Tian et al. (2025) showed that aligned LLMs remain trapped in a “safe attractor basin” created during RLHF, limiting exploratory novelty even with strong creativity prompts. There is a counterpoint from Jiang et al. (2024) showing LLMs can combine existing concepts in ways rated more novel than ideas from NLP researchers, but that is combinatorial novelty (recombining known things), not extrapolative novelty (ideas that break the frame).

The Pipeline Problem

This connects to something I was reminded of while looking at Karpathy’s LLM Wiki pattern and rohitg00’s v2 extension. A wiki is basically a collection of datapoints (documents, website, databases etc) that can be connected based on theme or subject. This leads to a great overview of data on a subject with beautiful graphs that can help you understand interconnectedness of the data points. However most wikis fail here: they compile raw sources into a structured, interlinked knowledge graph at ingestion time, resulting in a more or less fixed but definitely 1 to low-dimensional description of connections. Before any question has been asked, before anything is known about what can safely be dropped, irreversible choices are made at the point of minimum information about what those choices should preserve. And the problem here is that an LLM makes those choices, easy but at the cost of massive information loss.

This is not just a gut feeling. Information theory has a name for it: the Data Processing Inequality. It says that processing data can only destroy information, never create it. If X goes through transformation T to produce Y, then Y cannot contain more information about the original source than X did. Every step in a pipeline can only lose or preserve, never gain. It is the same discipline as keeping the RAW file instead of the JPEG, or lazy evaluation in programming: defer the irreversible reduction until the moment you know what you actually need.

The wiki does the opposite. It flattens at ingestion, the earliest possible moment, when it has the least information about what any future question will require. When the target artifact is a node-edge graph, only graph-shaped information survives extraction. Argument structure, evidential weight, mechanism, scope conditions, the reasoning connecting premise to conclusion: gone. Not because the model failed, but because it succeeded at building exactly the thing it was aimed at, and that thing has no room for any of it. A wiki link (even a typed one like “refutes”) collapses a multi-dimensional relationship into a single undifferentiated connection. The real relationship between two claims has independent components: logical relation, evidential strength, domain, temporal validity. The graph has one slot.

For planning a holiday this does not matter, losing some nuance about hotel reviews may not be so important (and you can deliberately choose that this is not important). But for anything where the quality of reasoning matters (medical research, legal analysis, policy decisions, engineering trade-offs), early information loss is crippling. For example three pro-articles enter a pipeline; the compiled page reflects the pro-standpoint and see it as the base stance. Then a fourth article is added with a very strong point contra, that should weigh much heavier than the three pro documents with light evidence. The model has no faculty for evaluating that the fourth article’s methodology is stronger. Evidential weight and citation count are different quantities, and the system only has access to the latter. Worse, the three pro-articles arrived first, so they created the concept pages, set the framing, and named things. The dissenting source arrives into a wiki already shaped to express the opposite position. It gets architecturally demoted to a status of a light contra, basically a footnote, where it should dismiss the entire pro standpoint.

And then it starts adding up: in a serial pipeline (entity extraction, graph building, contradiction resolution, consolidation), each step consumes the previous step’s output as ground truth. Chain a 95%-reliable step ten times and you end up with a reliability of 60% (0.95^10 –> 0.60). And on top of that these errors are correlated toward plausibility and majority framing, so the real number is worse. This is well-documented in cascading ML systems: Sculley et al. (2015) at Google showed that ML pipelines accumulate “technical debt” where each model’s errors become the next model’s training signal, creating feedback loops that are extremely hard to detect or debug. LLMs are slightly different but it is reasonable to state that they are not better at this.

Making it extra complicated if you are not a domain expert is that perceived reliability and actual reliability diverge in opposite directions as the pipeline grows. Each added step makes the surface richer, more complete-looking, more authoritative (more scores, typed edges, structure) while simultaneously multiplying against the reliability floor. The more elaborate the system, the more trustworthy it looks and the less trustworthy it (most likely) is.

The correct approach to choosing where to cut information density is as late and as informed as possible. The cut should be applied at the query, shaped by the query, discarding only what that specific question does not need.

And this connects directly to the novelty problem. Every mechanism in these systems (confidence scoring, contradiction resolution, consolidation, forgetting curves) is tuned to suppress exactly the signature of genuine novelty: low support, high conflict, poor fit with existing structure. A good new idea is an anti-consensus object. The pipeline converts whatever novelty does appear into “noise to be cleaned up,” because it has no faculty to see it as anything else.

Semi-Intelligence

So we can define what we are actually dealing with: full competence within the convex hull of expressed human thought, zero reach outside it.

This has quite some implication: most practical work lives inside that hull: writing code, synthesising research, drafting documents, debugging, refactoring, transferring methods from one domain to an adjacent one. LLMs are quite good at all of this. I use them for it every day and they make me measurably more productive.

But “inside the hull” has a boundary, and everything interesting about the future of ideas happens at or beyond that boundary.

The Great Equaliser

There is a way to frame this that makes the stakes clearer: the LLM is a nivellator. It pulls everyone toward the median of existing thought.

If you were below that median, you get lifted up. That is genuine value. I wrote about this before: the democratisation of capability is real and good. Someone who could not write code can now build a working application. Someone who could not write a decent email can now produce clear, professional communication.

But if you were above that median, or thinking orthogonally to it, the LLM pulls you down toward consensus. Your unusual angle gets smoothed into the conventional take. Your weird framing gets normalised. Your instinct that the consensus is wrong gets buried under a fluent, well-structured argument for why the consensus is right. The tool does not argue with you. It just quietly steers everything toward what the training data says is most likely.

The net effect is convergence toward the mean. The better the model gets, the stronger the pull. More people producing more competent work, all of it sounding increasingly alike. Ask five people to use an LLM to write a strategy document and you get five documents that could have been written by the same person.

If you are building systems on top of LLMs (and I am), be honest about what they can and cannot do. They are navigation tools for existing knowledge, not idea generators. Do not expect an LLM pipeline to surface the insight that changes your approach. It will give you a very competent synthesis of what is already known, and that is genuinely valuable, but it is a different thing.

The people who matter most here are the ones the LLM is worst at emulating: the imaginary thinkers, the artists, the scientists, the outliers, the stubborn contrarians who look at the consensus and say “but what if that is wrong.” People whose judgment is not a statistical shadow of what has already been thought, but a faculty that can value the unfamiliar as such. They were always important, and in a world where the median is increasingly automated, they are irreplaceable.

So hurray to the unconventional thinkers: the world needs you, so keep thinking those original thoughts.

Thinking about what LLMs can and cannot do for your team? Get in touch to compare notes.

Sources

On LLM creativity and limitations

• Has the Creativity of Large-Language Models Peaked? (Chakrabarty et al., 2025) – LLM-generated ideas lead to homogenous outcomes across domains
• Roll the Dice & Look Before You Leap (Tian et al., 2025) – Aligned LLMs stay trapped in RLHF’s “safe attractor basin”
• Neither Stochastic Parroting nor AGI (May 2025) – LLMs extrapolate from training priors but only within domain boundaries
• LLMs Can Realize Combinatorial Creativity (Jiang et al., 2024) – Counterpoint: combinatorial novelty is real, but distinct from extrapolative novelty
• Learning in High Dimension Always Amounts to Extrapolation (Balestriero et al., 2021) – The convex hull problem in high-dimensional learning

On information loss and LLM pipelines

• Data Processing Inequality – Information theory: processing can only destroy information, never create it
• Hidden Technical Debt in Machine Learning Systems (Sculley et al., 2015) – Google’s analysis of cascading errors in ML pipelines
• LLM Wiki pattern (Karpathy, 2026) – Compiled knowledge as alternative to RAG
• LLM Wiki v2 (rohitg00, 2026) – Extension with confidence scoring, temporal decay, and knowledge governance

Related posts

• When LLMs Actually Deliver – Where LLMs shine: the interior of the hull
• Why DIY Software Is Great Until It Is Not – Similar theme: tools change what is possible, but not what requires expertise
• Let the AI Pick: React – The conventionality filter in action: every LLM picks the same stack

Most people I know put wallpaper up in their own house, do small to medium jobs in and around the house themselves and only hire out to a crafts(wo)man when the job is too big or takes too much time. But doing stuff yourself is more or less normalised: it saves money and is fun to do. The same is now true for software engineering: with AI and vibe coding, anyone can build something that works.

That is a genuinely great thing. The DIY movement is a democratising development that put real capability into the hands of ordinary people, created an entire industry around it, and gave millions of people the satisfaction of building something with their own hands.

I also had the pleasure of seeing some really experienced carpenters work up close (30+ years). An interesting observation I made is that they do not seem to work very hard at all. However when you come back an hour later, real progress is made. No movement is wasted, every step is known and the whole workflow is optimised to minimise waste of resources and time. They know in advance how much time a certain job will take, what they need to do it and thus what it will cost.

I have also friends that bought old, badly maintained houses and turned them into small palaces in a few years. There were some hurdles to take and a lot of lessons learned, but the result is beautiful.

So what is the difference? One is a professional, the other is not. But what does that mean? Professional in my book was always someone who got paid to do a job, an amateur could do the same but was not paid for it. However there is a deeper distinction to be made: the diy-er builds for themselves, while the professional builds for others (and gets paid for it)

When evaluating the advent of AI and the impact it has on software engineering you can probably already see the parallel I want to make. AI is a great tool that helps the diy-er to build something really nice, perfectly adapted to what they want. However building for others (professional software engineering) has other demands.

The Software Version of This Story

AI-assisted coding is doing for software what the bouwmarkt (home improvement store, for non-Dutch readers) did for building and home improvement. Not just the tools, but the tools and the advice and the materials, all in one place. An IDE or a linter is a power tool: it makes you faster at what you already know. AI is the whole store: you can walk in knowing nothing and walk out with a plan, the supplies, and enough confidence to get started. And I think that is great.

I wrote about this before: the democratisation of software creation is genuinely exciting. The vibe-coded app that tracks your running schedule or manages your recipe collection or automates your invoicing? Build it. Have fun. Who cares if the code is messy. It works for you, and that is all that matters.

But then the app works so well that a friend wants it. Then a few more people want it. Someone suggests charging for it. And suddenly you are no longer building for yourself.

This is where it gets hard, and it is not about whether you can technically build the software. It is about everything else. When you build for others, you are working with someone else’s idea of what the product should do, not yours. You are responsible for their data, their money, their trust. You have to think about edge cases you would never hit yourself, because your users will hit all of them. You have to maintain it when you would rather be building something new. You carry financial risk if it breaks. You might need to comply with regulations you have never heard of.

More and more I get requests from diy-ers (non-developers, most of them in a company context) that ask me to evaluate their vibe-coded application. Most of the time the functionality is great (for their purposes), but there are a lot of open ends that need to be addressed.

Just as a professional carpenter would do when assessing a job, you need the expertise and the know-how (which goes far beyond the code itself, just as the carpenter knows far more than how to use a handsaw to saw wood) to analyse the situation and decide what is the best course of action: build on top of the application or tear it all down and start with a proper foundation.

It Is Not About Skill

I have made the transition from diy to professional twice. Before software, I worked as a sound engineer. That also started as a hobby: years of messing around with equipment, doing small gigs, learning by doing. The moment I went professional, the job changed. It was no longer about what sounded good to me, it was about what the client needed, on deadline, within budget, with backup plans for when things went wrong. The technical skills were the same, but everything around them was different.

Then I did it again with software. I had been programming for over twenty-five years for myself before making the switch to professional software engineering. The code I wrote as a hobbyist was fine for what it was. But writing code that other people depend on, that needs to run reliably, that handles their data responsibly, that complies with regulations I had never thought about was another game.

So this is not diy being bad and professionals good. Plenty of professional software is terrible and plenty of hobbyist software is brilliant. The distinction is not about ability.

The distinction is about context. When you build for yourself, you optimize for one person’s needs, and you can cut every corner you want because you are the only one who bears the consequences. When you build for others a lot of this changes:

• It is not your use case anymore, so you have to understand problems you do not personally have
• You carry responsibility for other people’s data and money
• You need to comply with rules that do not apply to personal projects (GDPR, accessibility requirements, industry regulations)
• Financial risk: if it breaks, the cost is not just your time
• You have to maintain it, even when you have lost interest

A professional software engineer does not just know how to write code. They know how to deal with all of that. How to handle errors that the happy path never reveals. How to design for concurrent users. Where the security boundaries are. What happens at 3 AM when the database fills up and nobody is watching. Not because they are necessarily smarter, but because building for others requires a different kind of thinking.

Where This Is Going

Look at how the construction industry works today. Nobody thinks it is strange that you wallpaper your own living room but call a carpenter for a kitchen renovation. Nobody thinks it is gatekeeping that you need permits and inspections when you build for others. DIY and the professional trades can both exist and are not mutually exclusive.

Software engineering is heading to the same place. People will keep building their own tools with AI, and that is good. But when the stakes go up, when it is someone else’s data, someone else’s money, someone else’s business, they will call in a professional. Not necessarily to write the code, but to assess what was built, identify the risks, and decide whether to build on top of it or start over with a proper foundation. Exactly the way a carpenter assesses a renovation job today.

The evaluation requests I mentioned are the beginning of that pattern. The role of the software engineer is shifting from “the person who writes the code” to “the person who knows whether the code is fit for purpose, and who understands the regulations, compliance, security, and data integrity requirements that come with building for others.” That is a bigger role which requires more expertise, because you need to understand both what was built and what should have been built.

AI will not replace software engineers. It will change what software engineers do.

Wondering whether your vibe-coded application is ready for others to use? Get in touch to find out.

Related Posts

• Vibe Coding: Product Quality and Democratisation – The product quality angle and the bottleneck shift
• Your AI Tests Are Probably Lying to You – One example of invisible quality: tests that pass without proving anything
• Human in the Loop: Why Your LLM-Assisted Code Still Needs Human Eyes – Why AI-generated code still needs professional review

So how do we measure this? User happiness: just ask for it, and measure some statistics. Security: be diligent and complete (and a bit obsessive), if needed do an audit. But what about maintainability? The goal is to make sure code is maintainable by adhering to a certain minimal standard, enabling developers and LLM’s to work faster, create better code and features more easily and create a better guardrail for code quality assurance.

In order to do that you need to know what works and what does not. What is a marketing hype? What is actually proven to work? Which methods of analysis can be trusted to use as a gate and which are at best an indicator? So I started by implementing what I already knew and some preliminary research which resulted in about 10 methods that I integrated into an mcp server: works fine, and it helps me tremendously to get a quick overview over a codebase, pointing out weaknesses. But at a certain point you keep adding tools and start wondering: do they actually add value to this analysis? The internet is full of tools (like CodeScene and SonarQube) that claim to help, but they all have a slightly different approach.

Given the increased complexity of my tool (and thus the analysis results) and that I plan to also implement the tool I build in CI/CD pipelines, I need to make sure that what I create makes some sense. So before writing any more code for that, I wanted to know: which code quality methods are backed by real evidence? Which ones sound scientific but are not? I get a lot of flagged problems, how to prioritise them? If some of those methods do not have enough grounding, I can delete them. And perhaps it will also tell something about paid options and their credibility and usefulness given costs.

I spent some time over the last months researching this, thinking this would be a solid field where methods would have proper evidence. What I found surprised me: the methods most tools focus on (complexity metrics, code smells, coupling scores) have weaker evidence than you would think.

This post is about the research, not the tools or the implementation. I want to lay out what I found so you can decide for yourself what is worth paying attention to.

Caveat: I am not an expert at code review methods (after all that is why I started this research). Please take it as such: I did my best to not miss any important papers, but if I have, please let me know.

What Is “Quality Code” Anyway?

Before diving into which measurement methods work, it helps to define what we are trying to measure, which is quite hard. Fortunately (or not entirely coincidental) my simple definition of what quality code is, resonates in the more official definitions.

Garvin (1984), adapted to software by Kitchenham and Pfleeger, identified five competing views of quality:

1. Transcendental: “I know it when I see it.” Recognizable but indefinable.
2. User-focused: fitness for purpose. Does it solve the problem? (Juran’s “fitness for use”)
3. Manufacturing: conformance to requirements and specifications.
4. Product-based: measurable through inherent properties (complexity, coupling, duplication).
5. Value-based: quality relative to cost. Different stakeholders weight it differently.

Most arguments about code quality are people talking past each other across these views. A SonarQube dashboard is view #4. A user who says “it works fine” is view #2. A manager asking “is it worth fixing?” is view #5.

The current international standard, ISO/IEC 25010 (2023), defines nine product quality characteristics: functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability, flexibility, and safety. That is a useful checklist, but we can not use it to measure the code directly. What matters for this post is the causal chain it formalizes, originally from McCall (1977) and codified in ISO 9126 (2001): internal quality (static code properties like complexity and coupling) influences external quality (runtime behavior like reliability and performance), which influences quality in use (actual user outcomes). Internal quality is necessary but not sufficient. Perfectly structured code that solves the wrong problem is still low quality. But you have to start somewhere, so let’s assume the external quality and usage quality are covered.

The measurement methods I evaluate try to measure internal quality through code properties (a.o. complexity, coupling, duplication, churn patterns) and use those as proxies for maintainability. They do not measure whether users are happy or whether the software solves the right problem. That is an important limitation to keep in mind: even if every metric below worked perfectly, it would only cover one dimension of quality.

The Setup

I looked at 13 different analysis methods that I either already built or considered building: hotspot analysis, code ownership, code smells, cognitive complexity, deep nesting, temporal coupling, duplicate code, coupling metrics, composite health scores, static analysis gates, function length, error handling analysis, and circular dependencies. For each one I collected supporting evidence and counterarguments from peer-reviewed papers.

These methods serve two different purposes. Some try to find specific bugs (static analysis catching a null dereference). Others assess code properties that make maintenance harder or easier (complexity, coupling, smells). Both matter for maintainability and code quality.

So for each method below I look at both: does it help find or predict bugs, and does it give useful insight about maintainability? Then I weigh whether it is worth keeping in a tool.

The Methods

Hotspot Analysis (Change Frequency x Complexity)

The idea: files that change often AND are complex are where your problems live. Nagappan and Ball (2005) showed that relative churn (normalized by component size) discriminated fault-prone binaries at 89% accuracy on Windows Server 2003 (I had that version running once, that is rather long ago). Raw commit counts are useless; you have to normalize.

Tornhill and Borg (2022) found that low-quality code (identified via hotspot analysis) contains 15x more defects and takes 124% longer to work on. This was across 39 proprietary codebases, peer-reviewed at IEEE/ACM TechDebt 2022. Tornhill is the founder of CodeScene, the tool that implements this approach. The research was done on CodeScene customers.

There is a deeper question though. The “complexity” half of hotspot analysis might just be measuring lines of code. Cyclomatic complexity, introduced by McCabe in 1976, counts the number of independent paths through a function: every if, for, while, or case adds one. The idea is that more paths means harder to test and more likely to contain bugs. It is probably the most widely used complexity metric in the industry. But Jay et al. (2009) showed that cyclomatic complexity has “absolutely no explanatory power of its own” beyond LOC. The correlation between code complexity and lines of code is so stable across languages and paradigms that they are effectively measuring the same thing.

Verdict: positive. Be aware the complexity half may just be measuring size, but enough evidence to actually use it.

Code Ownership and Knowledge Distribution

Bird et al. (2011) studied Windows Vista and Windows 7 and found that components with many low-expertise contributors had significantly more defects. Removing ownership features from their prediction model dramatically decreased performance, confirming that ownership is a genuine signal, not a proxy for something else.

This has been replicated across multiple Microsoft products and open-source projects. Thongtanunam et al. (2016) found that code review partially mitigates the ownership effect but does not eliminate it.

The counterargument: shared files may be inherently harder (more complex, more integration-heavy), and that is why many developers touch them. Correlation, not causation. And bus factor (the number of people who would need to disappear before nobody understands a piece of code) is a lagging indicator. By the time you measure that only one person knows how the billing module works, you are already in trouble if that person leaves.

Verdict: positive. Strongest evidence alongside hotspot analysis. Also a direct maintainability signal: knowing where knowledge is concentrated tells you where onboarding is most critical and where your team is most vulnerable.

Cognitive Complexity

SonarSource’s cognitive complexity (the metric behind SonarQube rule S3776) is probably the most widely deployed complexity metric today. And it does correlate with perceived understandability. Munoz Baron et al. (2020) confirmed this, and Lenarduzzi et al. (2023) found it slightly outperforms cyclomatic complexity for readability.

However it has not been validated as a defect predictor. “Harder to understand = more bugs” sounds logical, but nobody has proven this for cognitive complexity specifically. And it inherits the same fundamental critique as cyclomatic complexity: Shepperd (1988) showed that CC is “based upon poor theoretical foundations” and is “no more than a proxy for, and in many cases is outperformed by, lines of code.”

So cognitive complexity tells you something about readability as a formula approximates it, not necessarily how a human developer perceives the code. And it is worth noting the difference: when Buse and Weimer (2010) had 120 human annotators rate code readability, that did correlate with fewer defects. But that is human perception, not a formula counting nesting levels and branch points. Cognitive complexity claims to approximate what humans perceive, but the evidence that it actually does so well enough to predict outcomes is surprisingly thin for something so widely adopted.

Verdict: positive. Weak as a defect predictor, but the maintainability value is clear. A function with a score of 47 is harder for a new team member to understand than one scoring 8.

Deep Nesting

Nesting depth (how many levels of if/for/while are stacked inside each other) consistently correlates with fault rates. Hatton (1997) identified it as one of the most important dimensions that account for defect variability. It is more reliable than function length as a standalone metric.

But it is confounded with function length: deeply nested code is usually in long functions. El Emam et al. (2001) showed that after controlling for size, many metrics lose significance, and nesting may be one of them. There is also no evidence that reducing nesting (via early-return refactoring, for example) improves outcomes. You can scatter logic across more exit points and reduce the measured nesting without making the code any clearer.

Verdict: positive. The bug-prediction evidence is weakened by El Emam’s finding that nesting may just be a size proxy. It does not predict bugs better than LOC does. But deeply nested code is usually in long functions, and long functions do have more bugs, so the correlation comes along indirectly through size. As a maintainability signal it is more defensible: deeply nested code is genuinely hard to follow regardless of whether it independently predicts defects. Threshold-based (flag anything beyond 3-4 levels) is more useful than continuous measurement. Good supporting signal, not a primary one.

Code Smells

“God class” and “long method” have consistent defect correlation across studies. Palomba et al. (2018) found that “the great majority of analysed research papers found a positive correlation between code smells and software bugs.”

El Emam et al. (2001) placed that into perspective 17 years earlier: out of 24 OO metrics examined, only 4 retained any relationship to faults after controlling for class size. Large classes have more bugs because they have more code. The smell may just be a proxy for size. (more code, more bugs –> makes sense)

Sharma and Spinellis (2018) surveyed the full landscape of smell research, and the evidence per smell type varies a lot. Feature envy (a method that uses another class’s data more than its own) shows weaker and less consistent results. Long parameter list has almost no empirical support. Duplicated code gets the most research attention but its link to defects is debated (see duplicate code below). Fowler’s original catalog includes more exotic smells (message chains, middle man, speculative generality), but Khomh et al. (2012) only found that smell-affected classes are more change-prone as a group. That broad finding does not tell you which specific smells drive the effect, and the exotic smells have neither defect evidence nor clear maintainability benefits.

As maintainability indicators, the well-evidenced smells have direct value. A god class is hard to modify regardless of whether it contains bugs. The change-proneness finding from Khomh supports this: smell-affected classes get modified more often, which means developers spend more time on them. That is a maintenance cost even if it never produces a bug.

Smell detectors tend to produce a lot of noise. Bessey et al. (2010), writing from Coverity’s experience analyzing billions of lines of production code, found that false positives kill adoption. Users prefer fewer true findings over wading through noise.

Verdict: positive for god class and long method, which have both defect correlation and clear maintainability value. Negative for exotic smells (message chains, middle man, speculative generality), which have neither proven defect prediction nor demonstrable maintainability benefits.

Temporal Coupling (Co-Change Analysis)

Files that consistently change together may have hidden dependencies that structural analysis misses. D’Ambros et al. (2009) found that change coupling correlates with defects across three large open-source systems. Canfora et al. (2014) found 64-93% of defects in classes with Granger-positive results.

But temporal coupling is noisy. Bulk refactoring, API changes, and rename commits produce false positives. Results are system-dependent: what works for one project may not work for another. It needs aggressive filtering (minimum commit threshold, maximum files per changeset) to be useful.

Verdict: positive. Unique signal that no other method provides (hidden dependencies), but noisy as a standalone predictor. For maintainability, knowing which files are secretly coupled is valuable for planning refactors. Filter aggressively, present as “hidden dependency finder.”

Duplicate Code

Juergens et al. (2009) found that 52% of clones were inconsistently changed and 15% of those inconsistencies caused faults. Bettenburg et al. (2012) found only 1-3% of inconsistent changes introduce defects. And Rahman et al. (2012) found that clones may actually be less defect-prone than non-cloned code, possibly because cloned code tends to be simpler boilerplate.

Verdict: positive. Not a defect predictor, but genuinely useful for maintainability: when you fix a bug in one copy and forget the other three, that is a real maintenance problem. The value is in reducing the surface area of future changes, not in predicting where bugs are today.

Coupling Metrics

Coupling measures how much one piece of code depends on other pieces. Robert C. Martin proposed a framework that counts incoming dependencies (how many other modules use this one) and outgoing dependencies (how many other modules this one uses) to calculate an “instability” score. Theoretically clean. But Al Dallal (2013) found “a lack of theoretical and empirical evaluation.” The older Chidamber-Kemerer metrics from 1994, particularly their “coupling between objects” metric (simply counting how many other classes a class is connected to), have much better empirical validation. Martin’s metrics add little over that simpler measure. And as we found before, coupling metrics might just be proxies for LOC.

Verdict: positive. Weak for defect prediction, but hard to dismiss as a maintainability indicator. A module with 30 incoming dependencies is risky to change because any modification can break 30 other places. That is not about bugs, it is about change impact. Simple coupling counts (CBO) are sufficient; the fancier frameworks add little.

Composite Health Scores

CodeScene assigns a 1-10 “code health” score. The general principle (combining multiple metrics beats a single metric) is well supported. And Tornhill and Borg’s 15x defect rate difference is real.

But what does “7.2 health” mean? Fenton and Pfleeger (1997) point out that combining things measured on different scales (readability, coupling, churn) into one number violates measurement theory. It is like averaging temperature, wind speed, and humidity into a single “weather score.” You get a number, and it looks scientific, but what does it actually tell you? It seems like simplification taken too far.

Verdict: negative. Neither a reliable defect predictor nor a useful maintainability signal on its own, because it hides which dimensions are actually suffering. The principle of combining signals is sound, but a single number is not actionable. Better to show the individual signals and let the reviewer decide.

Static Analysis Quality Gates

Static analyzers scan source code without running it, looking for patterns that are known to cause problems. Like using a variable before it is initialized, a null pointer dereference, a SQL query built from unsanitized user input, a resource opened but never closed. Tools like SonarQube, ESLint (JavaScript), Pylint (Python), SpotBugs (Java), and CodeQL (GitHub’s query-based static analyzer) all fall in this category. Most work by matching code against a database of known-bad patterns, though some (like CodeQL) do more sophisticated data flow analysis.

77% of projects use at least one according to Beller et al. (2016). The economic argument is that it is cheaper to fix early, even with low recall. Many teams use them as quality gates in CI/CD: the build fails if the analyzer finds issues in new code.

But how low is that recall? Habib and Pradel (2018) found that static bug detectors miss the large majority of bugs. Different tools are mostly complementary, each finding different things. A clean report means nothing was found within that tool’s capabilities, not that the code is correct. Dijkstra (1969) said it decades ago: testing (and analysis) can show the presence of bugs, never their absence.

Verdict: positive. This is the one method that actually finds specific bugs rather than measuring code properties. Low recall, but what it catches is real. The economic argument holds: even catching 10% of bugs early is cheaper than finding them in production. Just do not pretend a clean report means the code is correct.

Function Length

Size correlates with total defect count. But it does NOT reliably predict defect density (bugs per line). And since LOC and cyclomatic complexity are linearly related, measuring both is measuring the same thing twice.

Verdict: negative. Does not predict defects beyond what LOC already tells you, and does not add maintainability insight beyond what complexity and nesting already capture.

Error Handling Analysis

Checking for bare except blocks, swallowed errors, ignored error returns (Go’s unchecked err), .unwrap() in non-test Rust code, await without try/catch. These are widely accepted as defects or at minimum bad practice.

There is no large-scale empirical study quantifying defect rates from specific error handling patterns. But the face validity is high: an ignored error return in Go is a bug waiting to happen, and the false positive rate is low when patterns are well-defined. The limitation is that these checks only catch absent error handling, not incorrect handling (wrong recovery action).

Verdict: positive. Limited academic validation, high practical value. Low false positive rate makes it safe to flag. Both a bug finder (swallowed errors are real bugs) and a maintainability signal (code that ignores errors is fragile).

Circular Dependencies

Circular imports make refactoring, testing, and deployment harder. Detection via DFS on the import graph is deterministic with zero false positives.

But there is limited empirical evidence directly linking cycles to defects. Small cycles may be benign. And this only addresses accidental complexity: fixing a circular dependency does not fix the underlying design problem that caused it.

Verdict: positive. Zero false positives makes it safe to always flag. Not a defect predictor, but a clear architectural health signal. A codebase with circular dependencies between major modules is harder to maintain and harder to test in isolation.

The Elephant in the Room: Everything Might Just Be LOC

This is the single most damaging critique in the field, and it applies to almost everything above.

El Emam et al. (2001) tested 24 object-oriented metrics and found that most lost their relationship to faults after controlling for class size. Jay et al. (2009) showed that cyclomatic complexity has “absolutely no explanatory power of its own” beyond lines of code. Graves et al. (2000) found that when LOC is included, complexity metrics add nothing to fault prediction.

The implication is if LOC predicts defects as well as your 47-metric dashboard, your dashboard is an expensive LOC counter with a nicer UI. Any metric that claims to be useful must demonstrate predictive power after controlling for size.

The Counterarguments That Apply to Everything

Beyond the LOC problem, there are critiques that cut across all methods. I collected them separately because they are worth reading as a group.

Goodhart’s Law. “When a measure becomes a target, it ceases to be a good measure” (Strathern, 1997). Developers split functions to hit a complexity threshold without improving readability. Code coverage targets lead to trivial tests. Any metric feeding into performance reviews will be optimized for the metric, not quality. (this I recognise from daily practice: a gate states that a function is too complex, so I rewrite it to pass the gate, but often that is a tradeoff on other areas)

The Halstead cautionary tale. Halstead (1977) proposed metrics based on operator/operand counts that were widely adopted in tools and standards. Then Shen et al. (1983) debunked them: conclusions based on sample sizes less than 10, core assumptions violated, conceptual errors. Metrics can be widely adopted and still be scientifically invalid. Adoption is not validation.

Models do not transfer. Briand et al. (2002) showed that fault-proneness models built on one project often do not transfer to other projects. Default thresholds in any tool are starting points, not universal truths.

False positives kill adoption. Bessey et al. (2010) again: users prefer fewer true findings over completeness. Their finding was that developers do not care about missed bugs nearly as much as they hate false positives. A false positive wastes your time right now, a false negative is invisible: you cannot be annoyed by a bug the tool never reported. After enough false alarms, developers stop checking findings entirely, and then even the true positives get ignored. Higher precision means some real issues slip through, but a tool nobody trusts has no use at all.

Only accidental complexity is addressable. Brooks (1986) distinguished essential complexity (inherent in the problem) from accidental complexity (artifacts of tools and languages). Code quality tools can only address the accidental kind. Design and requirements are where most difficulty lives. No tool changes that.

DeMarco’s nuance. Tom DeMarco claimed “you cannot control what you cannot measure” in 1982. In 2009 he revisited that position, writing that his earlier work “made the suggestion that metrics are good and therefore more metrics would be better.” He did not reject metrics entirely, but argued that the most important software projects are transformational, and transformation cannot be measured or controlled in advance.

So What Do I Do With All This?

Looking at all 13 methods through both lenses (bug finding and maintainability), a pattern emerges. Hotspot analysis and code ownership have the strongest evidence for predicting defects, and they are also directly useful for maintainability. Static analysis actually finds bugs, even if it misses most of them. Everything else (complexity, smells, coupling, duplicates, nesting) is weak as a defect predictor but has varying degrees of value as a maintainability signal.

This is not a complete overview, not by far. There are methods I did not cover (architecture-level metrics, test quality indicators, dependency freshness, among others) and areas where the research is evolving faster than I can read it. I may revisit this in follow-up posts.

Coming back to paying for services that do this for you and whether or not they are adding value: the strongest signals come from git history, which is free in itself. The metrics most paid tools emphasize are the ones with weaker defect-prediction evidence. You are often paying for convenience, visualization, and CI integration rather than for better science. That can be worth it, but go in knowing what the science actually supports. From maintainability standpoint there is a good case to use them, and also it saves you from having to build this yourself (even though that equation dramatically changed with LLMs at your disposal). But it is much more fun to understand what happens and build it yourself (even if you do not use it for you production code, you’ll learn a lot).

I liked this quote from DeMarco in his article from 2009 so let’s end with that: “Software development is and always will be somewhat experimental. The actual software construction isn’t necessarily experimental, but its conception is. And this is where our focus ought to be. It’s where our focus always ought to have been.”

Looking into code quality improvement for your team? Get in touch to compare notes on what works and what does not.

Sources Mentioned

• No Silver Bullet (Brooks, 1986) – Essential vs accidental complexity
• A Critique of Cyclomatic Complexity (Shepperd, 1988) – CC is a LOC proxy with poor foundations
• Software Measurement: A Necessary Scientific Basis (Fenton & Pfleeger, 1997) – Textbook on measurement theory in software
• Predicting Fault Incidence (Graves et al., 2000) – LOC makes complexity metrics redundant
• The Confounding Effect of Class Size (El Emam et al., 2001) – Most metrics are LOC proxies
• OO Metrics Don’t Transfer (Briand et al., 2002) – Models built on one project fail on others
• Software Science Revisited (Shen et al., 1983) – Debunking Halstead metrics
• Use of Relative Code Churn Measures (Nagappan & Ball, 2005) – Churn predicts defects at 89% accuracy
• Software Engineering: An Idea Whose Time Has Come and Gone? (DeMarco, 2009) – “More metrics is better” recanted
• Change Coupling and Defects (D’Ambros et al., 2009) – Change coupling correlates with defects
• Cyclomatic Complexity and Lines of Code (Jay, Hale et al., 2009) – CC has no explanatory power beyond LOC
• Code Clones in the Large (Juergens et al., 2009) – Inconsistent clone changes cause faults
• A Few Billion Lines of Code Later (Bessey et al., 2010) – Coverity’s lessons on static analysis adoption
• Learning a Metric for Code Readability (Buse & Weimer, 2010) – Human readability ratings correlate with defects
• Don’t Touch My Code! (Bird et al., 2011) – Code ownership predicts defects
• An Empirical Study on Clones (Bettenburg et al., 2012) – Only 1-3% of clone inconsistencies cause defects
• Do Code Smells Hinder Code Changes? (Khomh et al., 2012) – Smell-affected classes are more change-prone
• Cloned Code: Stable Code (Rahman et al., 2012) – Clones may be less defect-prone
• A Validation of Martin’s Metric (Al Dallal, 2013) – Martin’s coupling metrics lack validation
• Defect Prediction with Change Coupling (Canfora et al., 2014) – Granger-based temporal coupling predicts defects
• Analyzing the State of Static Analysis (Beller et al., 2016) – 77% of projects use static analysis
• Code Review and Ownership (Thongtanunam et al., 2016) – Code review partially mitigates ownership effect
• A Survey on Software Smells (Sharma & Spinellis, 2018) – Evidence per smell type varies widely
• On the Diffuseness and Impact of Code Smells (Palomba et al., 2018) – Smells correlate with bugs (with size caveats)
• How Many of All Bugs Do We Find? (Habib & Pradel, 2018) – Static analyzers miss the large majority of bugs
• Cognitive Complexity Validation (Munoz Baron et al., 2020) – Validated for readability, not defects
• Code Red: The Business Impact of Code Quality (Tornhill & Borg, 2022) – 15x more defects in low-quality code
• Cognitive Complexity vs Cyclomatic Complexity (Lenarduzzi et al., 2023) – Cognitive complexity slightly outperforms CC for readability

Tools Referenced

• SonarQube – Most widely used static analysis platform
• CodeScene – Behavioral code analysis (churn + complexity + ownership)
• ESLint – JavaScript/TypeScript linter
• Pylint – Python static analyzer
• SpotBugs – Java bug pattern detector
• CodeQL – GitHub’s query-based static analyzer
• Adam Tornhill, Your Code as a Crime Scene – The book behind CodeScene’s approach

Related Posts

• Your AI Tests Are Probably Lying to You – Similar theme: green dashboards that hide real problems
• Benchmarking Open-Source PII Detection – Same approach: benchmark tools, pick the practical winner
• Vibe Coding: Product Quality and Democratisation – Code quality in the AI era
• Human in the Loop: Why Your LLM-Assisted Code Still Needs Human Eyes – Code review quality with AI assistance
• Evidence-Based Best Practices as AI Guardrails (Part 1) – Same evidence-based methodology applied to AI guardrails

As you may have read in an earlier post, I do not like writing tests. So when an LLM is able to write it for me: great, I let Claude do it. In a review a colleague pointed out that two of my tests did nothing more than verify that mocks return what mocks are told to return. The tests passed, the ci passed but it mainly tested a very obvious thing. And also useless. I read them myself three or four times and did not see it either.

That was seven months ago and it has been nagging me: how to improve tests while not spending too much time on it. After all I do rely on tests, so they have to have a certain quality. But I do not want to have to write them myself.

Some more observations

After that happened I started looking more carefully at AI-generated tests in general. Not just mine, also in code I reviewed for colleagues, and I saw some patterns: Tests that assert a mock returns what you told it to return. Tests that wrap the assertion in a try/except so they pass even when the check fails. Tests with if status == 200: assert ... that silently skip when the status is something else. Etcetera.

So I went looking at what the academic world has to say about this. Of course this has been researched: “rotten green tests”, coined by Delplanque et al. in 2019. What it means is tests whose assertions never actually execute. The concept of “test smells” goes back even further, to van Deursen et al. in 2001 and Meszaros in 2007. So this has been a known problem for 25 years. What is new is that LLMs produce these smells at industrial scale. (Then again the LLM’s training data probably contains at least some of these badly formulated and constructed tests)

LLM is not so good at generating tests

I spent some time going through recent studies on LLM-generated tests and the results are not encouraging.

One study tested LLMs on unit test generation across 17 Java projects. Between 34% and 62% of generated tests did not even compile. Of the ones that did compile and run, 75% of the bugs they missed were missed because the tests used boring, normal input values. The LLM picks safe values that exercise the happy path even when the code is broken. If you need to set a value to NaN or pass an empty string or hit an exact boundary to trigger the bug, the LLM will not do that. It generates the test equivalent of “hello world.”

Another survey of 115 publications on LLM test generation found that the best model they evaluated detected 8 out of 163 real bugs (0.74%).

And CodeRabbit’s analysis of 470 open-source PRs found that AI-co-authored code has 1.7x more issues overall and up to 2.74x more security vulnerabilities than human-written code.

Those numbers are quite high (higher than I recognise from my daily practice, but that has partly to do with the guardrails I use on every LLM interaction).

The real problem is that you can not trust your tests: LLMs often filter out failing tests before showing you results. They throw away the tests that fail (which might be the ones that actually found something) and present you tests that pass. And then teams look at “400 tests, all passing” and feel confident, which is not a confidence that is justified.

One option is to build the tests first (TDD), which works quite well and I do implement that (partly) by defining tests before the actual code is written. However I also found that the building process can only be planned to a certain extent: chaos is always looming (a package that does not work as expected, incomplete view of how the database works etc.) So while I do believe that TDD is a nice idea, I do see more productive progress in a slightly less defined and rigid way of TDD. But however you want to do this: back to the main question: how to make sure that tests are of a proper quality and test things that matter and do not test things that do not matter?

So I Built a Thing

The idea is simple: instead of a generic “review this code” prompt (which misses test smells because it focuses on code quality, not test purpose), give the LLM a specific taxonomy of test defects to hunt for. And crucially, make it read both the test file AND the production code the test covers. You cannot judge whether a test proves anything without seeing what it is supposed to be testing. So after some research I defined several categories that are recognisable to an LLM. See the resources section below if you want to know more.

The skill checks for nine categories:

• Rotten green tests – tests that pass without verifying what they claim. Tautologies, conditional assertions, swallowed errors
• Classic test smells – Assertion Roulette (multiple asserts, no messages), Eager Test (one test doing too much), Mystery Guest (depends on invisible external state)
• Mock abuse – so many mocks that the test is not testing real behavior anymore. If your test has more mocks than assertions, something is off
• Contract drift – tests that verify internal method call order instead of observable output. These break on every refactor even when behavior stays the same
• Missing negative paths – no 401 test, no 403 test, no input validation test. Every auth-protected endpoint needs these
• LLM-generated defects – hallucinated APIs that do not exist in the codebase, generic-only inputs, type-not-value assertions like isinstance(result, dict)
• Fixture fragility – tests that depend on execution order, shared mutable state, or fixtures that do too much
• Assertion quality – loose assertions (len(x) >= 1 when exact count is known), missing assertions, or ten checks on the happy path and one on the error path
• Missing edge cases – empty collections, boundary values, concurrent access, cleanup after mutation

You run it like this:

/review-tests                          # review changed test files on current branch
/review-tests backend/tests/unit/      # review a specific path

Output is as small as possible, with one line per finding:

tests/test_auth.py:L42: bug: assert mock.return_value == mock.return_value. Tautology. Assert against expected value from spec.
tests/test_api.py:L18: risk: try/except catches AssertionError. Test passes on failure. Remove bare except.
tests/test_search.py:L91: gap: No 401 test for /api/search endpoint. Add unauthenticated request test.

The Contradiction: LLMs can create bad tests and find them quite well

Santana Jr et al. evaluated LLMs on detecting test smells in Java projects and found detection rates up to 96% for classic smells. LLMs are better at finding test smells than rule-based tools, especially for things like Assertion Roulette and Eager Test that require understanding what the test is trying to do.

The same technology that mass-produces rotten green tests is also the best tool we have for finding them. I noticed this in practice too. Claude is most of the time pretty good at writing tests, but “pretty good” is not good enough when you need tests that catch real bugs. Left to its own devices it gravitates toward happy paths and shallow assertions. But when you give it a specific list of things to look for and force it to read the production code alongside the test, it catches things I miss. Not always, not perfectly, but consistently better than my own manual review.

Where I Use It

I run it when i decide it is useful (large amount of tests) and as part of my automated PR review. The skill is part of codebench, where I collect Claude Code skills, hooks, and configuration. If you use Claude Code for test generation (and you probably do, because writing tests is boring), give it a try.

Letting AI write your tests and wondering what slips through? Get in touch to compare notes.

Resources

Research

• Refactoring Test Code (van Deursen et al., 2001) – The original test smells paper
• xUnit Test Patterns (Meszaros, 2007) – 68 patterns for maintainable tests
• Rotten Green Tests (Delplanque et al., ICSE 2019) – Tests that pass because their assertions never execute
• Growing Object-Oriented Software, Guided by Tests (Freeman & Pryce, 2009) – Test behavior, not implementation
• Investigating Severity Thresholds for Test Smells (Spadini et al., MSR 2020) – Not all test smell instances are equally harmful

LLM Test Generation

• On the Evaluation of Large Language Models in Unit Test Generation (Yang et al., 2024) – 34-62% of LLM-generated tests fail to compile; 75% of undetected defects due to missing triggering inputs
• Large Language Models for Unit Test Generation (Alshahwan et al., 2025) – Survey of 115 publications; best model found 8 of 163 bugs
• Evaluating LLMs Effectiveness in Detecting and Correcting Test Smells (Santana Jr et al., 2025) – LLMs detect test smells at up to 96% accuracy
• State of AI vs Human Code Generation Report (CodeRabbit, 2025) – AI-co-authored code has 1.7x more issues and 2.74x more security vulnerabilities

Tools

• review-tests skill – The Claude Code skill discussed in this post
• codebench – Skills, hooks, and configuration for Claude Code
• AI Regression Testing skill – Community skill for AI blind spots in test generation

Related posts

• Human in the Loop: Why Your LLM-Assisted Code Still Needs Human Eyes – The meta-test story that started this
• From Prototype to Production: Automated Builds – Where automated test review fits in the build pipeline
• Benchmarking Open-Source PII Detection – Similar approach: benchmark tools, pick the practical winner

About seven months ago I wrote about zero-latency PII filtering in Python logging: how to sanitize personal data from logs without blocking your main thread. That solved the where and when of filtering. But it implied that detection itself was a solved problem. Throw some regex at it, done. And when handling logs, this is somewhat easier, since you can define certain keys to be filtered etc.

However filtering PII in text is not a solved problem. When I started planning to build PII redaction into a production pipeline that handles legal documents and Dutch-language content (part of a broader push toward security by design in our projects), the question became: which detection tool actually works across all of that?

So first thing to do is get your bearings: find the most common approaches and some datasets and run a benchmark. Five detection approaches, four datasets, two languages (Dutch (since this was the language for the legal documents) and English (testing this will give some more insight and means I can reuse what I learn here)). This post covers what I found. A follow-up (part 2) will take the winners and push them further: more entity types, domain-specific tuning, the stuff you actually need for production.

Warning: this is not a scientific paper. It is a practical comparison I ran to figure out which tool to bet on for our pipeline. The difficulty with comparing these tools is that they are fundamentally different: different architectures, different label schemes, different assumptions about what PII even is. There is no clean apples-to-apples comparison possible. I did my best to make it fair (shared label set, multiple datasets), but four datasets and five models is not enough for strong statistical claims. What it is enough for is picking a direction.

Why Comparing PII Detection Tools Is Hard

Comparing PII detection tools is getting complicated veryfast. Every tool uses different labels, supports different entity types, and has different ideas about what counts as PII.

Piiranha outputs fine-grained types: GIVENNAME, SURNAME, CITY, STREET, BUILDINGNUM. Presidio outputs coarse types: PERSON, LOCATION. GLiNER accepts whatever labels you define at inference time. A “person name” in one system is two separate entities in another.

Coverage gaps make it worse. Piiranha detects 17 PII types but has no DATE entity. Presidio has DATE_TIME but cannot detect passwords or usernames. No two tools cover the same set.

And every dataset has its own taxonomy. AI4Privacy uses GIVENNAME/SURNAME. Gretel uses “name.” Nemotron uses first_name/last_name. CoNLL-2002 uses PER.

A naive benchmark that ignores all this is misleading. A model scoring F1 0.63 on one dataset and 0.13 on another might not have “collapsed.” It might simply lack entity types that dataset emphasizes (dates accounting for 25% of spans, for example).

Three Questions People Conflate

When evaluating PII detection, three separate questions get mixed together:

1. How good is the model at detecting entity types it supports?
2. How many entity types does it support?
3. How well does it generalize to text from different domains?

Evaluating all entity types on a single dataset mixes all three. A model might score poorly not because detection is bad, but because it lacks entity types the dataset emphasizes. And a model that scores well might just be evaluated on data similar to its training set.

I wanted to isolate question 1 first. So I defined a shared label set of six entity types that all models can detect: PERSON_NAME, EMAIL, PHONE, LOCATION, CREDITCARD, and IBAN. Every model gets scored only on these. Fair fight.

Question 3 gets answered by testing across four independent datasets. Question 2 (which additional types each model uniquely supports) is left for part 2.

The Models

Five detection approaches, covering the major architectural categories:

Regex: Compiled patterns for email, phone (via Python phonenumbers for international format parsing), IBAN, credit card (with Luhn validation), and IPv4. Deterministic, no ML.

Piiranha: 86M-parameter DeBERTa-v3-base, fine-tuned for token classification on 17 PII entity types. Trained on AI4Privacy PII-Masking-300k. Supports English and Dutch.

Presidio (Microsoft, v2.2): Orchestration framework combining spaCy NER (en_core_web_lg / nl_core_news_md) with built-in pattern recognizers for structured entities.

GLiNER v1: ~209M-parameter zero-shot NER model. Entity types are defined as natural-language descriptions at inference time. You tell it what to look for, and it looks. Schema-agnostic. This is the GLiNER architecture fine-tuned on synthetic PII data.

GLiNER v2: 205M-parameter multi-task model from Fastino Labs. A separate project from v1 with different architecture and training. Included because v2 claims improved multi-task capabilities.

The Datasets

Four datasets, two languages, different domains:

The combination is imperfect. No single dataset covers both languages with non-synthetic, human-annotated PII spans. As far as I can tell, such a dataset does not exist publicly.

Evaluation

Accuracy: Precision, recall, and F1 with relaxed span matching (+/-5 character tolerance). Micro-averaged across all entity types.

Speed: Median latency per text and throughput in texts/sec. Measured on Apple Silicon M1 Pro CPU.

Reversibility: Detected spans get replaced with typed placeholders ([PERSON_NAME_1], [EMAIL_1]). Same entity text gets the same placeholder throughout a document. Pass rate = percentage of documents where restore(redact(text)) == text. This matters: if you cannot perfectly restore the original after redaction, your redaction system is lossy and you will corrupt data.

Results

Cross-Dataset Accuracy

The first thing that jumps out: none of these models are good. The best average F1 across four datasets is 0.542. Commercial systems claim 0.92-0.99. That is a big gap.

The second thing: the top three (Piiranha, GLiNER v1, Presidio) are closer to each other than any of them are to “good enough.” The difference between first and third place is 0.061 F1. A paired t-test across the four datasets shows none of the pairwise differences reach statistical significance (p»0.10, n=4). They are all mediocre, just in different ways. Piiranha swings wildly (0.169 to 0.780). GLiNER v1 is more consistent (0.455 to 0.607). Presidio sits in between.

Generalization

How much does each model degrade on unfamiliar text?

Piiranha’s 78% drop on financial text looks dramatic, and it is. But GLiNER v1’s “consistent” performance means it never goes above 0.607 either. Presidio’s drop is a moderate 17%. The variance differs (Piiranha std=0.271, GLiNER v1 std=0.077, Presidio std=0.213), but all three models land in the same general territory of “not good enough for production without additional work.” Regex scores zero on CoNLL-2002 because that dataset only has person names and locations.

Person Names (The Hard Part)

Person names are the most common PII entity and the hardest to detect because they are context-dependent (is “Holland” a person or a location?).

Piiranha excels on synthetic data (AI4Privacy, Nemotron) but fails hard on formal financial text and is mediocre on newspaper text. GLiNER v1 and Presidio are more stable but with lower peaks. Regex cannot detect person names at all.

Phone Numbers (Format Matters)

GLiNER v1 is the most consistent phone detector across all domains. GLiNER v2 is significantly worse than v1 here. Presidio struggles because spaCy was not designed for this.

Email (Where Patterns Win)

Presidio’s pattern-based email recognizer beats all NER models. This is one of the few entity types where pattern matching outperforms learned models. Emails have rigid structure, and a well-written regex does not need to “understand” context. The transformer models still do well on AI4Privacy and Nemotron, but drop on Gretel’s financial documents where email formats are embedded in formal text.

Speed

Presidio is 8-11x faster than transformer models (spaCy’s backbone is small). Among transformers, Piiranha’s 86M parameters make it fastest. All measured on CPU. GPU may change the picture, but the relative speed differences will probably be comparable.

Reversibility

Presidio fails on 36% of documents. The cause: its internal tokenization produces character offsets that do not precisely match the original text. This will require further work but can be improved.

Does Regex Help?

A reasonable improvement could be to add regex to the mix, so I tested this. Both regex and the model are running on the same text and the results are merged.

Conclusion: adding regex to NER is not worth it. The largest gain is +0.012 for Piiranha, and even that is driven entirely by the Gretel dataset where Piiranha is already failing. For GLiNER v1 and Presidio, the improvement is within noise.

Regex actually hurts accuracy on AI4Privacy for all three models (-0.010 to -0.011 F1) because the merge logic occasionally displaces correct NER detections with regex false positives. The model choice matters far more than layering regex on top.

So Which One Do You Pick?

If the accuracy differences are not significant and none of the models are production-ready on their own, what does differentiate them?

Piiranha and GLiNER v1 are standalone models. They give you detections and that is it. You build the anonymization pipeline, the recognizer registry, the deanonymization logic, the operator pipeline, the multilingual engine configuration, all of it yourself.

Presidio gives you most of that out of the box. Recognizer registry, operator pipeline for anonymization and deanonymization, built-in pattern recognizers for entity types the NER models lack (DATE_TIME, URL), multilingual NLP engine support, and active maintenance by Microsoft. Its accuracy gap to the top two is 0.054-0.061 F1, which is not statistically significant with our sample size.

And to keep in mind: Presidio’s NER backend is replaceable. Its architecture is designed for plugging in custom recognizers. You are not hard bound to just spaCy. You can plug in GLiNER v1, Piiranha, or anything else as a custom recognizer and possible get better detection quality while keeping the framework.

When accuracy is equally mediocre across the board, the framework wins. Presidio is the logical choice. Not because it detects PII best (it does not), but because it provides the infrastructure you need regardless of which detection model you use, and you can swap the detection model later without rebuilding everything else. This is a very practical benefit over the others.

The reversibility issue (36% failure rate) is real but can be improved. It is caused by span boundary misalignment in Presidio’s tokenization, not by its detection quality. Post-processing Presidio’s output to align spans with the source text before replacement solves this.

Speed is another Presidio advantage. At 15ms per text it is 8-11x faster than the transformer models. For batch processing this matters less, but for real-time pipelines it is significant.

What I Learned

The main takeaway: none of these tools are good at cross-domain PII detection. Not the open-source ones I tested, and not the commercial ones either. Tonic.ai reports F1 0.92-0.99 on their own corpus. AWS Comprehend claims 0.87-0.91. OpenAI’s Privacy Filter hits 0.96 on PII-Masking-300k. But these are all in-distribution numbers. When PIIBench tested eight systems across 10 unified datasets with 48 entity types, the best system scored F1 0.14. OpenAI’s Privacy Filter drops from 0.96 to 0.18-0.65 on Tonic’s out-of-distribution test groups. Cross-domain PII detection is genuinely unsolved.

So in the end the decision stops being about which model detects best, since hey are all mediocre. The decision becomes: which tool is the most practical to build on, the easiest to use, the most actively maintained, and the most complete? That is Presidio. It gives you the framework, the pipeline, the pattern recognizers, and a swappable NER backend so you can improve detection quality over time without rebuilding everything else.

A few other things worth noting:

In-distribution benchmarks are misleading. Piiranha’s F1 of 0.78 on its training data drops to 0.17 on financial text. If a model’s benchmark only shows results on familiar data, that number is not useful as a predictor for real-world use.

Zero-shot models generalize better than fine-tuned ones. GLiNER v1 beats Piiranha on every out-of-distribution dataset. Its schema-agnostic architecture seems to help. This makes it a strong candidate for plugging into Presidio as a custom recognizer.

Hybrid regex+NER is oversold. Common recommendation in the literature (including RECAP and other hybrid approaches). Our results show it adds almost nothing (+0.001 F1 for GLiNER v1).

GLiNER v2 is worse than v1 for PII. The multi-task rewrite hurt NER quality. v2 underperforms v1 on three of four datasets.

What This Benchmark Does Not Prove

This is a practical comparison to pick a direction, not a rigorous evaluation. Here is what is wrong with it and why I ran it anyway.

Three of four datasets are synthetic. AI4Privacy, Gretel, and Nemotron are all machine-generated text. CoNLL-2002 is the only real-world data, and it is Dutch newspaper text from 2000. So when I say “cross-domain,” I mostly mean “across different synthetic data generators.” The gap between synthetic financial text and actual financial documents could be larger than the gap between two synthetic datasets. I used what was publicly available with character-level PII span annotations in both English and Dutch. That combination barely exists. A proper evaluation would include real legal documents, real customer support transcripts, real medical records. Unfortunately they are out of reach to test this on.

AI4Privacy is Piiranha’s training distribution. I use the validation split, not the training split, so it is not literal data leakage. But validation data from the same generator is not independent. Piiranha’s 0.78 on AI4Privacy is inflated compared to what it would score on truly held-out data, and the “78% drop to Gretel” headline is partly an artifact of that inflated baseline.

The +/-5 character tolerance is a judgment call. Span matching in NER evaluation is sensitive to boundary definitions. Strict matching (exact character positions) penalizes models for including or excluding a space, a period, or a title like “Mr.” Relaxed matching with a 5-character window allows for these tokenization differences without being so loose that partial detections count as hits. Five characters is roughly one token or one word boundary. I picked it because it felt right for this use case, so it was a choice without real base. Other benchmarks use token-level overlap or IoU thresholds. The choice affects absolute F1 numbers but not the relative ranking between models (all models benefit equally from relaxed matching).

CoNLL-2002 only tests two of six shared entity types. It has person names and locations but no email, phone, credit card, or IBAN. Presidio’s 0.780 on CoNLL-2002 is driven entirely by spaCy’s person/location NER, not by its pattern recognizers. If CoNLL-2002 had structured PII entities, Presidio’s score there would likely be higher (its email recognizer is the best in the benchmark) and the other models’ scores might shift too. This dataset tests a subset of the shared label set, not the full thing.

No per-entity sample counts. I do not report how many instances of each entity type appear per dataset. If Gretel has 12 credit card numbers and AI4Privacy has 400, the per-entity tables are not equally powered. A model scoring 0.139 on person names in Gretel might be based on hundreds of spans or dozens.

Six shared entity types is a narrow view. In practice you need more: dates of birth, social security numbers, IP addresses, Dutch BSN numbers. Piiranha detects 17 types. GLiNER can attempt any type you define (with varying accuracy). The choice depends not just on detection quality but on which entities you need. Part 2 covers this.

All that said: I needed to pick a tool and waiting for a perfect benchmark could take a long time. In the end my question was answered in the sense that there is no ‘best’, just different flavours. And in that light choosing Presidio is the practical choice.

What Comes Next

The direction is set: Presidio as the framework, with a better NER model plugged in.

Part 2 will:

1. Test Presidio with GLiNER v1 and Piiranha as custom recognizers to see how much the detection quality improves while keeping the framework
2. Evaluate OpenAI’s Privacy Filter, which claims F1 0.96 on PII-Masking-300k but drops significantly on out-of-distribution text according to Tonic.ai’s benchmark. It is relatively new and worth testing on our datasets
3. Evaluate on the full set of entity types we actually need for production: person names, emails, phone numbers, physical addresses, dates of birth, social security numbers, IBANs, credit cards, IP addresses, and Dutch-specific identifiers (BSN)
4. Fix the reversibility issue (span boundary alignment)
5. Determine where we still need to augment with additional detection methods

Building PII detection into your pipeline and wondering which tool to pick? Get in touch to compare notes.

Resources

• PIIBench: A Unified Multi-Source Benchmark for PII Detection - Cross-domain benchmark where all tested systems scored below F1 0.14
• Tonic.ai: Benchmarking OpenAI’s Privacy Filter - Independent cross-domain evaluation showing commercial systems degrade significantly on out-of-distribution text
• AWS Comprehend Detect PII Service Card - AWS’s own accuracy claims per entity type
• OpenAI Privacy Filter - F1 0.96 on PII-Masking-300k, much lower cross-domain
• GLiNER: Generalist Model for Named Entity Recognition (Zaratiana et al., NAACL 2024) - The architecture behind GLiNER v1
• Hybrid Rule-based NLP and Machine Learning for PII in Financial Documents (Nature Scientific Reports, 2025)
• RECAP: Hybrid Methods for Multilingual PII Detection (NeurIPS, 2025)

Models and datasets tested

• Piiranha v1 - DeBERTa-v3-base fine-tuned for PII detection
• GLiNER v1 (multi_pii) - Zero-shot NER fine-tuned on synthetic PII data
• GLiNER v2 (gliner2-base) - Multi-task NER from Fastino Labs
• Presidio - Microsoft’s PII detection and anonymization framework
• AI4Privacy PII-Masking-300k - Mixed synthetic PII dataset (EN + NL)
• Gretel Synthetic PII Finance - Multilingual financial documents
• Nemotron-PII - NVIDIA’s multi-industry PII dataset
• CoNLL-2002 - Dutch/Spanish NER with human annotations

The Pitch

I have been iterating on automated builds for over a year now. Different tools, different approaches, lots of starting over. This evolved into a Go server with a React frontend that could orchestrate Claude Code containers to plan, build, and review code autonomously. It ran on my local server and it worked. I tested it on my own and a few work repos and the output was good: decent quality code and a lot of time saved.

When I showed this to the team, the reaction was “how soon can we plug this into our workflow?” The appeal was straightforward: we can do more with the same team, product owners and domain experts can contribute ideas directly, and we are less dependent on developers for every small change.

Used to Jira

The prototype proved the concept but it was not a fit for company use. It replaced too much: it had its own project management UI, backlog and task board. Nice for my own use, but we already have Jira. We are used to it (not addicted to it), so it seemed a good choice not to change everything at once: do not replace Jira. Everything still works without the LLM layer. Issues get created in Jira the normal way. Developers can pick them up the normal way. The automated builder is an addition, not a replacement.

This matters because nobody had to change how they work on day one. A product owner creates a Jira issue, adds it to the sprint, and if they put it in a specific status column with a specific label the builder picks it up. If the ticket does not have the label, a human developer picks it up instead. Same board, same workflow, one extra option.

I can imagine we move away from Jira eventually. But now is not the right moment. We need to gain experience with this step first, and forcing a tool change at the same time would muddy the results.

How It Works

The system is called Codebuilder for internal use. It polls Jira for issues in a trigger status, spawns Docker containers running Claude Code, and reports results back as comments, status transitions, labels, and pull requests.

Jira (issue moves to trigger status)
  |
  v
Poller --> Engine --> Docker Container
                        |
                        v
                    Claude Code + build script
                        |
                        +-- calls back to Codebuilder API
                        +-- uses MCP server for project knowledge + Jira
                        +-- creates draft PR on GitHub
                        |
                        v
                    Container exits --> Engine completes job
                        |
                        v
                    Jira comment + status transition

The worker inside the container does planning and building in one pass. It has access to project knowledge (specifications, requirements, architectural decisions) through an MCP server, and it can read and update the Jira issue it is working on. When it is done, it creates a draft PR. A developer reviews, possibly makes small changes, and merges. The merged PR auto-transitions the Jira issue to done.

Half an hour from “issue in queue” to “PR ready for review.”

What It Looks Like

In essence there are five steps that run in a single docker container, every step is a new claude code session:

• research
• plan
• build
• review
• fix

The reason why everything runs in docker containers is mainly security: this isolates claude code from the server, making sure the LLM (which runs unsupervised), has no access to the server. Also we make sure no secrets are available (like a .env file), to minimise risks. This does mean that small errors can be part of the build, but this is acceptable over allowing a very knowledgeable but non-responsible entity roaming around your files.

What we added: Azure AD single sign-on, Axiom for structured logging, Telegram notifications, CSRF protection, rate limiting, and a CLI tool for developers to review and merge automated PRs from their terminal that integrates with the mcp server to make sure github and jira get updated through the cli as well.

Another important one: product owners and others can interact with Jira through Claude Code directly in the browser. We already have the setup to run Claude Code on the server, so by exposing that through a web interface someone can use the full setup without leaving their browser.

This helps tremendously with refining issues and keeping the project up to date. Nothing nicer than asking “what small issues and bugs are there on the backlog that we could combine in one run?” without having to go through a long list yourself.

The Bigger Picture: Not Just for Developers

The part I am most excited about is what this enables for people who are not developers.

The idea: a product owner or domain expert fills out a form describing what they want. That goes into a refinement queue. An LLM (with human oversight) helps refine the idea into a well-scoped issue with acceptance criteria. The refined issue either goes to the automated build queue or gets assigned to a human developer, depending on complexity and judgment.

This is not “non-technical people writing code.” It is non-technical people contributing meaningful input to the development process without needing a developer to translate their ideas into tickets. The LLM handles the translation, a developer or product owner reviews it, and the builder does the implementation.

What Worries Me

Two things, and they are the same ones from my previous post.

Token costs. The quality is high because we run rigorous planning and review cycles, both LLM and human. But the planning and review steps require the best models to maintain a minimum quality bar. Cheaper models produce reviews that miss subtle issues or flag correct code as broken, and bad automated reviews are worse than no reviews because they create false confidence. Token prices keep shifting, and not always downward. That uncertainty makes it hard to forecast costs for the next quarter. In the end it is still cheaper than a purely human review, but the gap is not as large as it used to be.

Developer fatigue. Reviewing LLM-generated code is fundamentally different from writing code yourself. You lose the mental map you build when you write the code. You are reading someone else’s work in a codebase you did not shape, and the entity that wrote it does not learn from your feedback across sessions. After a few hours your attention drops. After a few days you start wondering if this is what the job looks like from now on. I wrote about this before and it is still true, just more visible now that multiple people experience it instead of only me.

Our current answer: all PRs get an automated LLM review as part of the workflow. The developer decides whether to also ask a colleague for a human review. The guideline is to always request one for database changes, large frontend or backend changes, and anything security-related. At first this seems a bit uneasy, not having a second reviewer. However, the automated review is so thorough that it catches more than most (if not all) humans would.

Where We Are Now

We are in the “use it carefully and learn” phase. It works, it is fast, the code quality is good, and it is cheaper than a developer hour. A year ago I started stitching together shell scripts to run Claude Code in sequence. Now a product owner can drop an idea into Jira and get a reviewable PR back before lunch.

There is a lot more to cover: how we handle failures, what the prompts look like, how we scope issues for the builder, token cost breakdowns, and what happens when the model gets it wrong. That will be a follow-up post once we have more production hours under our belt.

Want something like this for your company? I can build it, and it is a lot of fun to do. Get in touch and we will figure out what fits your workflow.

Resources

• Claude Code - Anthropic’s agentic coding CLI
• Model Context Protocol - The protocol Codebuilder uses to expose project knowledge to workers
• Axiom - Structured logging and observability
• Azure AD OIDC - Authentication setup

  Related posts

These cover the individual steps and lessons behind Codebuilder in more detail:

• The Orchestrator: Automating Full Claude Code Workflows - The earlier orchestration approach: research, plan, build, review chained together
• When LLMs Actually Deliver - The tooling that makes LLM builds produce usable output
• Human in the Loop: Why Your LLM-Assisted Code Still Needs Human Eyes - Why automated review alone is not enough
• Fully Automated LLM Builds: Where It Actually Stops - Costs, fatigue, and model ceilings at scale
• Why I Shrunk Claude Code’s Context Window Back to 200k - Context management lessons that feed into how workers are configured
• Supercharge Claude Code with Custom Configuration - The agent/skill/hook setup that Codebuilder workers inherit

This morning I watched a video on context window management in Claude Code as part of my daily “keep up with what is happening in the LLM space” routine. Good content, solid diagnosis of the problem. But everything in it was about manual interventions: trigger compaction at the right moment, use structured handoffs, rewind instead of correcting. All valid techniques. But there is an issue that is buried deeper and there are two settings, buried in the documentation, that may solve most of this.

The Problem With More Room

Since Opus 4.6 it ships with a default 1M token context window. Five times the 200k window of their predecessors. Sounds like a pure upgrade. Great!

It is not. The single most important thing for working with LLMs is context management: keep it as small as possible with as relevant info as possible and nothing more than that. Anthropic’s engineering team says you should be “striving for the minimal set of information that fully outlines your expected behavior.” Their documentation acknowledges that “as token count grows, accuracy and recall degrade, a phenomenon known as context rot.” The root cause is the n-squared attention mechanism: double the context, quadruple the number of pairwise relationships the model has to track.

I am not alone in experiencing this. Some quick research finds similar sentiment: Paul Gauthier, the creator of Aider, found that “every model seems to get confused when you feed them more than ~25-30k tokens.” He calls it the number one problem his users report. JetBrains Research tested observation masking (hiding old tool outputs) and found a 52% cost reduction while boosting solve rates by 2.6%. Less context, better results. The NoLiMa benchmark found that 11 of 12 tested models dropped below 50% of their short-context performance at just 32k tokens. Not 200k. Not 1M. 32 thousand.

In practice this means: more hallucinations, forgotten instructions, goal drift, inconsistent decisions. Not at 900k tokens. Much, much earlier.

What I Keep Seeing

A lot of the advice I come across focuses on manual interventions. Trigger compaction yourself at the right moment. Use a new session or /clear when switching tasks. Save state to a JSON file before clearing. Ask Claude for periodic summaries. Use sub-agents to keep intermediate work out of your main context.

These are all valid. I use sub-agents heavily (they get their own fresh context window, which is the single most effective architectural pattern for avoiding context rot) and /clear between unrelated tasks. But manual interventions are workarounds for a window that is too large, not fixes for the underlying problem. They require you to watch your context usage while trying to get work done. That is overhead the tooling should handle.

Amp went further: they dropped compaction entirely and designed around short threads with clean handoffs. Their senior engineer Dan Mac put it bluntly: “You should basically never use compaction.”.

The Simpler Fix

Two environment variables solve this without ongoing attention:

CLAUDE_CODE_DISABLE_1M_CONTEXT set to 1 caps the context window back to 200k tokens. It removes the 1M model variants from the model picker entirely.

CLAUDE_AUTOCOMPACT_PCT_OVERRIDE set to a value between 1-100 controls when auto-compaction triggers, as a percentage of context capacity. The default is around 95%, which means on a 1M window, compaction does not kick in until you are at 950k tokens. That is way past the point where quality has degraded.

Set both in your project or user settings:

{
  "env": {
    "CLAUDE_CODE_DISABLE_1M_CONTEXT": "1",
    "CLAUDE_AUTOCOMPACT_PCT_OVERRIDE": "70"
  }
}

200k window at 70% threshold means compaction triggers around 140k tokens. Well before quality drops off. Compaction runs more frequently but with less context to summarize, which means better summaries. Andrej Karpathy described context engineering as “filling the context window with just the right information for the next step.” Too much, and “performance might come down.” A constrained window forces that discipline automatically.

The Cost Angle

Every turn in Claude Code sends the full conversation context to Anthropic’s servers. If your context window is sitting at 600k tokens of accumulated tool output, file reads, and old conversation, all of that gets re-sent and re-billed on the next message.

With Opus 4.6 at $5 per million input tokens, a 600k context costs a lot per turn just for input (not exactly 3 dollar because there is also caching going on). A 140k context (right before compaction) theoretically costs $0.70. Over a long session with dozens of turns, that difference adds up. One user on Hacker News described burning through $100 in credit when Opus 4.6 “got stuck in a bullshit reasoning loop.” So a smaller window is better for quality and is cheaper.

Compaction Is Not a Safety Net

The standard advice is to rely on compaction to keep your context clean. And compaction works, sort of and sometimes. It summarizes the conversation to free up space. But it is lossy. The model decides what matters and what gets dropped, and its judgment is not always yours. Often I feel like I have to start over again with the nuances of the problem we were working on.

Another problem is what happens after compaction. Yesterday I was in a session, working on changes across a repo. Auto-compaction kicked in mid-task. First thing Claude did after compacting: committed everything. Without being asked. It lost enough context to forget that I had not asked for a commit, saw uncommitted files and went ahead.

Thariq Shihipar from the Claude Code team recommends compacting proactively at 50-60% capacity instead of waiting for auto-compaction. Good advice. But if you constrain your window to 200k and set the threshold to 70%, you get roughly the same effect automatically. No need to watch your token count and manually trigger /compact at the right moment.

There is another upside to the smaller window: compaction itself gets better. If compaction triggers at 70% and reduces back to roughly 30%, the 1M window has to summarize away 400,000 tokens of conversation. The 200k window only discards 80,000. Five times less information to lose. Smaller contexts lead to more accurate summaries. (For the information theory purists out there: I am aware that it is more complicated than this, please forgive my shortcuts)

Fresh Sessions, Not Long Ones

The env vars help within a session. But the bigger win is avoiding compaction by not having long sessions in the first place.

My workflow separates every phase into its own session. Research runs in one session, planning in another, building in a third. Never chained together in the same conversation. I built an orchestrator that does this automatically: each step launches a separate Claude Code instance. The whole rationale is context isolation. Each phase starts clean with only the information needed to start that session.

This is the same principle behind sub-agents, just at a larger scale. When Claude Code spawns a sub-agent, that agent gets its own fresh context window. All intermediate work (file reads, grep output, failed attempts) stays in the sub-agent’s context. Only the final result comes back. Morph’s research found a 90% performance gain using sub-agent architecture over single-agent. The reason is straightforward: every file read and tool call that stays out of your main context is noise that never competes for attention.

The Counterintuitive Takeaway

The 1M context window is a capacity increase, not a quality increase. More room means more space for noise, higher bills, and worse output once you cross the degradation threshold. Steve Smith calls it “a huge junk drawer.” Glen Rhodes describes context as working memory, not storage, and argues you should treat it like RAM on a constrained system: deliberate about what gets loaded, suspicious of anything that lingers.

The best results I get from Claude Code come from keeping the window small, avoid compacting at all, and never letting one phase pollute the next. Two environment variables and a habit of starting fresh. That is the whole trick.

Running into context issues or managing your own Claude Code setup? Get in touch to compare notes.

Resources

Claude Code documentation

• Using Claude Code: session management and 1M context – Thariq Shihipar’s practical guide to context management
• Effective context engineering for AI agents – Anthropic’s engineering blog on context rot and mitigation
• Claude Code environment variables – Official docs including the two env vars discussed here
• Explore the context window – Interactive simulation of how context fills during a session
• Context windows API docs – Model-by-model context sizes and Anthropic’s acknowledgment of context rot

Research and analysis

• Lost in the Middle (Liu et al., 2023) – The foundational research on performance degradation in long contexts
• Context Rot research by Chroma – Evaluation of 18 LLMs showing universal degradation with input length
• JetBrains: Smarter Context Management for Agents – Observation masking: less context, better results
• Morph: Context Rot complete guide – Sub-agent architecture and agent-specific context data
• Compaction research across coding tools – Claude Code, Codex CLI, OpenCode, Amp compared

Developer perspectives

• Paul Gauthier on practical context limits – Aider creator: models get confused above 25-30k tokens
• Karpathy on context engineering – “Too much or too irrelevant, and performance might come down”
• Amp drops compaction for handoff – Why one coding tool designed around short threads
• Why Context Windows Won’t Keep Growing Forever – Steve Smith on diminishing returns and the junk drawer effect
• Context as working memory, not storage – Glen Rhodes on treating context like constrained RAM

Related posts

• The Orchestrator: Automating Full Claude Code Workflows – Each phase in its own Claude Code instance
• Fully Automated LLM Builds: Where It Actually Stops – Token costs as a ceiling on automation
• gtk: Filtering CLI Noise to Save Tokens – Reducing what goes into context in the first place

This weekend I built a small CLI tool that pulls transcripts, comments, and metadata from YouTube videos. The first thing I fed it was David Soria Parra’s keynote “The Future of MCP” at the AI Engineer conference. David wrote the original Python MCP SDK at Anthropic, so he knows where the protocol is heading. I dove into MCP about 9 months ago as part of the exploratory phase of a government job, a lot has happened since, and there is a lot on the roadmap. I discussed it with an LLM, asking about REST’s role, about playbooks that instruct models how to compose tools, and about the similarities with what I have been building.

What David Laid Out

The short version: 2025 was about coding agents (local, sandboxed, verifiable). 2026 is about general knowledge workers who need connectivity to five SaaS apps and a shared drive, not a local compiler. He sees three layers for this: skills (domain knowledge in files), MCP (rich semantics, auth, governance, long-running tasks), and CLI/computer use (great when the tool is already in pre-training data like git or gh). The best agents will use all three.

Three things he wants the ecosystem to fix: progressive discovery (stop dumping all tools into context, load them on demand), programmatic tool calling (give the model an execution environment to compose multiple calls in one script instead of round-tripping one by one, like Cloudflare’s Code Mode does), and designing for agents, not REST (stop mapping REST endpoints 1:1 into MCP servers, he called conversion tools “cringe”).

The upcoming features add a lot, but the new feature I care about most is skills over MCP: servers shipping domain knowledge alongside their tools. More on that below. The protocol is barely 18 months old, with 110 million monthly downloads (roughly 2x faster than React hit that number), so that is proving how popular it is.

I Already Built This

The part that clicked with me was “skills over MCP.” David described it as an upcoming protocol primitive: servers should ship playbooks alongside their tools, instructing the model how to combine them for specific tasks. The server author maintains the playbooks, not the user. When workflows change, the server updates its skills and every connected agent gets the new instructions automatically.

I have been doing exactly this in logbench, the MCP server I built for querying our Axiom logs. Tools like explore_dataset don’t return raw data. They return step-by-step instructions: “first get the schema, then run an error breakdown, then drill into the top categories.” The model picks the right workflow tool, gets the recipe, follows it. (Not entirely my idea, did something similar a long time ago, but this step was inspired by Axiom’s official mcp code)

It works well. No context bloat because the playbook only loads when the model calls that specific tool. Progressive discovery is built in for free. And the instructions are scoped to the task at hand, not a generic “here are all the things you could do.”

Skills Over MCP: The Distribution Angle

Before I get to why formalization worries me, there is one part of skills over MCP that is genuinely exciting: distribution.

Right now, if I want my colleagues to use the logbench playbooks, they need my exact MCP server setup. If I want to share a highly specialized tool behind an auth wall or a paywall, there is no standard way to do that. Skills over MCP solves this. Your team connects to the same MCP server and everyone gets the same playbooks, updated by the server author, no local configuration needed. A specialized log analysis skill, a compliance checking workflow, a financial reporting recipe: all distributed through the same protocol, access controlled at the server level.

That is a real improvement over “copy this markdown file into your project.” It means you can build tools that are genuinely sharable across teams, organizations, even commercially. The distribution story is strong.

The Boring Toolset Problem

But here is where I am less enthusiastic. What I see happening with MCP is the same thing that happens to every successful protocol: it moves from “wow, that is cool” to the inevitable boring enterprise toolset, the same kind of standardization convergence I wrote about with React. Mediocre-good-for-all, mostly optimized for large organizations with compliance requirements, not for developers who want to push boundaries.

My concern is not that the formalization itself will limit what I can do. It probably will not. My concern is what happens to developers along the way. When I built the playbook pattern in logbench, I understood exactly what was happening: a tool returns instructions, the model follows them. I learned how to engage with the model, how to structure instructions it would follow reliably, what worked and what did not. That understanding came from building it myself, from prodding and experimenting. Once that becomes a protocol primitive you just consume, the experimentation stops. You get a standard way to do it, and most developers will never look underneath.

That is how we lose the skill of working with LLMs directly. Not because the abstractions are bad, but because they are comfortable. People stop experimenting with how to instruct models, how to structure tool interactions, how to design playbooks that actually work. They use the MCP skills primitive because it is there, and they never discover new patterns that only emerge when you build from scratch.

MCP itself is open source now (Anthropic donated it to the Linux Foundation as part of the Agentic AI Foundation), which is good. But the ecosystem it lives inside is moving in a direction I like less. Claude Code started as a developer-focused tool, the kind of thing where you could wire up your own workflows and push the boundaries. Increasingly it is becoming a fits-all product, and the pricing reflects that. The whole Claude Code environment is powerful, but it is also an ecosystem that wants you to stay inside it.

Which is why I think it is worth looking at what exists outside. Pi is one framework worth trying for agentic use, approaching connectivity differently from MCP. There are more options than the one path Anthropic is paving, and the best time to explore them is now, while the patterns are still forming and nothing is locked in.

What Happens to REST?

This is the question I kept coming back to. As more and more interaction moves through agents, and agents interact through MCP, what happens to the classic API?

REST is not going anywhere as plumbing. You cannot have MCP without REST (or something like it) underneath. The comments on the talk pushed back hard on this point, and they are right: MCP is essentially “discoverable REST,” and the move toward stateless transport is literally re-converging toward REST patterns.

But I wonder about the trajectory: right now we have API-centered systems and we are moving toward API + MCP. Will that become MCP-centered? Eventually MCP-only for some use cases? And if so, what happens to the APIs that remain?

I think they change character, the classic REST API is a developer’s tool: full CRUD, every resource exposed, every operation available, everything must be in there. The MCP model is different: only those operations that add value, combined into higher-level actions when that makes sense. Less “here are all the building blocks” and more “here is what you can actually do.”

That is not a developer-centered design: it is a human-centered design, or an LLM-centered design, which turn out to be surprisingly similar. And if agents become the primary consumers of APIs, the APIs that stick around will probably start looking more like MCP tools than like the CRUD interfaces we build today. Fewer granular endpoints, more intent-oriented operations. Domain knowledge shipped alongside the API, not buried in documentation.

Or put differently: if your API is so granular that you need a playbook to use it, perhaps the API itself should be the playbook.

What I Took Away

Watch the full talk if you work with MCP or build agent tooling. It is 18 minutes and dense with where the protocol is heading.

My takeaways, for what they are worth:

• The playbook-as-a-tool pattern works today, no protocol extension needed. If you are building MCP servers, try it before waiting for the formalized version.
• Progressive discovery is not optional at scale. If you dump 50 tools into the context window, you are doing it wrong.
• MCP is a good protocol, but keep building things yourself too. The understanding you get from direct experimentation with LLMs is worth more than any abstraction.
• Look beyond Anthropic’s ecosystem. Try Pi, try building without MCP, see what works. The best patterns come from exploration, not from consuming frameworks.
• As agents become primary data consumers, the APIs themselves will loose importance and start looking more like MCP tools: fewer CRUD endpoints, more intent-oriented operations.

Building MCP servers or thinking about API design for agents? Get in touch to compare notes.

Resources

• The Future of MCP – David Soria Parra keynote at AI Engineer conference
• Model Context Protocol specification – the official MCP spec and documentation
• FastMCP – the Python SDK David called “way better” than the official one
• Cloudflare MCP server and their Code Mode blog post – example of exposing an execution environment instead of individual tools
• Agentic AI Foundation announcement – Anthropic donating MCP to the Linux Foundation
• David Soria Parra on GitHub
• When LLMs Actually Deliver – my earlier post on logbench and the playbook pattern