A certain level of confidence is what breaks things. It’s more akin to the quiet confidence of someone who has used a tool so frequently that they no longer question how it operates than arrogance. Any serious AI lab today will have open-plan offices, standing desks with dual monitors, and whiteboards still covered in chalk from the most recent sprint review. You’ll find intelligent people working on models at a remarkable rate. People who can rigorously explain the behavior of those models mathematically will be harder to come by.
The gap between what we can create and what we truly understand is beginning to appear less like a small issue and more like the main issue facing technology today.
The scope of the situation is difficult to overlook, according to the Stanford AI Index for 2026. In about three years, AI has spread to 53% of the world’s population, surpassing both the internet and personal computers in speed. In one way or another, 88% of organizations have embraced it. These figures point to infrastructure rather than a trend—the kind of ingrained reliance that makes it extremely difficult to reverse when something goes wrong.
However, the theoretical framework supporting all of this is, to put it simply, inadequate. Nowadays, industry, not academia, produces about 90% of frontier AI models. The level of transparency has decreased. Training data is becoming more and more confidential. Researchers who used to publish in-depth theoretical analyses are now employed by businesses where publication is a liability to competitors. With each product cycle, it seems as though the formal understanding of what these systems are actually doing—in a mathematical, provable sense—is getting farther behind.
The surface results are so impressive that it is difficult to talk about. On PhD-level scientific tasks, AI now performs on par with or better than humans. Benchmarks for coding get close to 100% parity. The demonstrations are truly amazing. However, the same systems continue to fail on about half of basic tasks, such as reading an analog clock, which highlights a crucial point: capability without theoretical foundation is uneven in ways that are genuinely difficult to predict. This has been dubbed the “jagged frontier” by Stanford researchers, and the term seems appropriate. The progression is not seamless. There is currently no trustworthy map of this network of peaks and sinkholes.
Recently, Peter Thiel made headlines when he noted that AI appears to be more detrimental to technical workers than to creative ones; in his words, “math people are more exposed than the word people.” It’s a controversial assertion that might be true in the context of the labor market. However, it suggests something more profound. Those who are most adept at using AI may not always be aware of its theoretical limitations. The most dangerous tech engineers aren’t those who disregard AI, but rather those who mistake comprehension for fluency, according to a piece that was posted on Medium a few weeks ago. Because it has the awkward texture of being true, that sentence has been making the rounds.
A related concern regarding technical debt has been brought up by MIT Sloan researchers. Technical debt is code written quickly with AI assistance that saves time now but causes compounding issues down the road. It’s possible that we’re incorporating the same dynamic into the field itself, making rapid progress on applied systems while covertly accruing conceptual debt that will eventually become due.
In the background is a mathematical problem that dates back a century. A few years ago, researchers from Cambridge published work showing that AI has intrinsic limitations related to a classical paradox—limits that are formal rather than merely practical. It’s not a moral failing that the majority of working engineers have never taken the literature seriously, but it does indicate that the field is developing on unexplored territory.
It’s still unclear if regular institutional pressure will be sufficient to close the gap or if something going horribly wrong will need to force the reckoning. As you watch this play out, you can’t help but notice that the same tension keeps coming up: confidence versus knowledge, speed versus comprehension, and application versus theory. The machines are picking up new skills. The question is whether those who are constructing them are also learning quickly enough.
