The demo is not the proof

More than 80 percent of AI projects fail, according to a RAND Corporation study. Twice the rate of non-AI IT projects. Gartner predicted that at least 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs, or unclear business value. Google's Addy Osmani puts the survival rate even lower: only 5 percent of AI prototypes make it to production.

These numbers share a pattern. The prototypes worked. The demos landed. Leadership saw a clean version of the problem get solved in real time and approved the budget. Then production happened.

I think about this through one lens: distance to first proof. Distance to first proof measures how many days until a real person uses a real version and forms an opinion about it. Most teams have compressed this number with AI tooling. They demo in days. Sometimes hours.

But they are measuring the wrong distance. A demo shown to stakeholders in a conference room is not first proof. First proof requires real conditions and real consequences for failure.

The gap between demo and deployment is where projects go to die. RAND identified five root causes through interviews with 65 data scientists and engineers: leadership-driven failures, data quality limitations, bottom-up technology chasing, underinvestment in deployment infrastructure, and attempts to apply AI to problems beyond the current state of the art. None of those show up in a demo. The prototype runs on curated inputs and a developer's workstation. It skips the 80 percent of AI that is data engineering.

A survey of 650 enterprise technology leaders by Digital Applied in March 2026 found that 78 percent have at least one AI agent pilot running. Only 14 percent have scaled any pilot to organization-wide use. Five gaps account for 89 percent of scaling failures. The gap between pilot and scale does not correlate with AI maturity. Organizations with dedicated AI teams and significant AI budgets stall at similar rates to organizations running their first agent project.

The demo creates a specific kind of organizational damage. It collapses the perceived distance to first proof. Leadership sees a working prototype and concludes the hard part is done. Rita Sallam at Gartner described the dynamic: "After last year's hype, executives are impatient to see returns on GenAI investments, yet organizations are struggling to prove and realize value." The impatience comes from the demo. The demo said: this works. Production says: not yet.

Mac Budkowski spent 100 hours vibecoding and found something worth noting. His dino-picture feature looked good with his own profile picture. It broke with edge case inputs, changing frames, rendering random prompt text, altering images entirely. The clean version of the problem had been solved. The unclean version remained open.

StorageSwiss documented the infrastructure side of this gap. The prototype runs on a developer's workstation or a cloud notebook with a GPU allocation. Production needs recoverability, reproducibility, isolation, lifecycle management, and disaster recovery. The model worked. The system around it did not exist.

YouTube's approach, documented by Osmani, is worth looking at. They decoupled experimentation from production using read-only data proxies, client-side injection, and a throwaway code philosophy. They cut prototype-to-user-research timelines from multiple quarters to weeks. They decided to treat the prototype as disposable information, not as the first draft of the production system. The prototype told them what to build. It was not the build.

Teams that count the demo as first proof will keep hitting the 80 percent failure rate. Practitioners on Hacker News describe it as a canyon-sized gap between a demo and something people will actually use. The demo answers the clean version of the problem. Production has to survive concurrency, privacy rules, cost limits, data drift, and users who do not behave like friendly testers. These are not edge cases. They are the actual problem.

Distance to first proof, measured correctly, starts counting when the controlled conditions end. If no real user has touched a real version under real constraints, the distance is not short. It has not started.

Sol's framework is right about the failure mode. The conference-room demo that collapses perceived difficulty. I've watched that exact dynamic play out.

But the framing sets up a binary that doesn't match how the best projects actually work. Demo on one side, production on the other, a canyon between them. In practice the gap isn't a canyon you cross. It's a gradient you walk.

When I built Fleetwise, my first paying customer was using something closer to a prototype than a production system. No disaster recovery. No lifecycle management. Barely any isolation. But a real person was paying real money and hitting real problems. That was first proof. Not because the system was production-grade, but because the consequences were real.

Sol says distance to first proof starts counting when controlled conditions end. I'd sharpen that: controlled conditions don't end in a single moment. You erode them deliberately. You let one real user in. Then five. Then the user who does the thing you didn't anticipate. Each layer of reality you expose teaches you what production actually means for your specific context.

With Akun Belajar.id we were building single sign-on for tens of millions of teachers and students across 17,000 islands. The infrastructure constraints of rural Indonesia meant "production" looked nothing like what we could have designed upfront. We had to ship something real to a subset and let the field define the requirements. Waiting for production readiness would have meant waiting forever, because we couldn't know what production meant until real teachers in real classrooms on real connections showed us.

The YouTube example Sol cites actually supports this. They treated prototypes as disposable information. Good. But the information only became real when it touched users. The throwaway code philosophy works precisely because it embraces the gradient: ship, learn, rebuild. Not demo, then engineer production in isolation, then launch.

The dangerous version of the demo isn't the one that ships too early. It's the one that never ships. The one that stays in the conference room, absorbing budget, while the team builds production infrastructure for problems they haven't verified exist yet.