TLDR: Challenger launched in 1986 because 24 missions had flown without incident, and past success was treated as proof of future safety. AI governance repeats that mistake when it treats deployment as the finish line. A deployed system needs an owner and ongoing monitoring for as long as it runs.
Table of Contents
In 1986, NASA launched the Space Shuttle Challenger on a morning when the temperature at Cape Canaveral had dropped to 28 degrees Fahrenheit. The O-rings sealing the solid rocket boosters had never been tested at that temperature. Engineers at Morton Thiokol, the contractor responsible, raised the concern the night before launch. They were overruled.
The decision to launch was made by people who had seen the system work many times before and concluded that prior success was sufficient evidence of future safety. The shuttle had flown 24 missions without incident. The governance framework said: if it worked before, it is approved to fly again.
Seventy-three seconds after launch, Challenger broke apart.
The failure was not in the design. It was in the assumption that a system, once validated, remains valid regardless of changing conditions.
AI governance is making the same assumption at scale.
Quick definition of terms
-
AI governance is the set of roles, rules and processes that decide how an AI system is built, deployed and checked.
-
Audit is a review of a system at one point in time.
-
Drift is when the data or conditions a deployed model meets move away from those it was validated on.
-
Post-deployment monitoring is ongoing tracking of how a live system behaves, with a named owner who acts on what it finds.
The One-Time Audit Problem
Most organizations that deploy AI systems treat governance as a pre-deployment activity. You evaluate the model, test for bias, document the outputs, satisfy the regulatory checklist, and ship. After that, the system runs. The world changes around it. The model does not.
Compliance teams are brought in after deployment rather than before, and when tough questions surface, ownership is often unclear. That pattern describes organizations that at least have compliance teams. Many do not. Ajithp
According to McKinsey’s 2024 State of AI, 72% of enterprises have AI systems in production, but only 9% describe their AI governance as mature. The gap between those two numbers is where things break.
What breaks, specifically, is the assumption that the conditions under which a model was validated are the conditions under which it will operate indefinitely. They are not. Training data ages. User behavior shifts. The context the model was built for drifts away from the context it is actually running in. The model keeps producing outputs. Nobody is checking whether those outputs still correspond to reality.
What Deployment Looks Like Without Ongoing Governance.
Jake Moffatt was traveling to Ontario after his grandmother died. He asked Air Canada’s chatbot about bereavement fares and was told he could apply for a reduced rate retroactively, within 90 days of purchase. He booked the flight, traveled, submitted the application, and was denied. The chatbot’s information contradicted Air Canada’s actual policy.
Lol
Air Canada argued it could not be held liable for information provided by its chatbot. The Tribunal found that Air Canada still bore responsibility for all information on its website, whether it came from a static page or a chatbot.
Air Canada’s defense was, in effect, that the chatbot was a separate entity responsible for its own outputs. The Tribunal described this as a remarkable submission.
What is actually remarkable is that the chatbot had been running with incorrect information, and nobody in the governance chain had caught it. The system was deployed, the audit was done, and the assumption was that the work was finished. It was not. The chatbot kept talking to customers. The policy information it carried kept being wrong.
This is a small case involving a few hundred dollars. The structural problem it illustrates is not small at all.
The Incentive Problem
The reason ongoing governance is rare is straightforward. CIOs get measured on deployment velocity, not governance maturity. Shipping AI features moves quarterly metrics, but building governance infrastructure does not.
Organizations track what gets them credited. Deploying a model gets credited. Monitoring it after deployment does not appear on any metric that determines budgets or performance reviews. So monitoring happens inconsistently, if at all.
Current frontier AI regulations primarily focus on systems deployed to external users, where deployment is more visible and subject to outside scrutiny. High-stakes applications that occur internally, where companies deploy systems within their own organizations to automate R&D, accelerate critical business processes, and handle sensitive proprietary data, face three specific gaps: scope ambiguity that allows internal systems to evade regulatory obligations, point-in-time compliance assessments that fail to capture the continuous evolution of internal systems, and information asymmetries that subvert regulatory awareness and oversight. Litigate
Point-in-time compliance is the Challenger problem restated. The system passed the test at launch. What happens after launch is a different question, and the current governance architecture largely does not ask it.
What Continuous Governance Actually Requires
The aviation industry figured this out the hard way. Aircraft are not certified once and flown indefinitely. They go through mandatory maintenance cycles, inspection requirements, and airworthiness directives that update the operating parameters when new information emerges. A fleet-wide safety concern discovered after deployment triggers a review of every aircraft in the fleet, regardless of whether any individual plane has shown a problem.
That architecture exists because the consequences of failure are immediate and visible. An aircraft that goes down produces evidence. The governance system is designed around that evidence loop.
AI failures are often neither immediate nor visible. A model that drifts produces outputs that are subtly wrong. Decisions made on those outputs produce outcomes that are hard to trace back to the model. The harm is distributed, delayed, and mediated through other systems. By the time it becomes visible, the causal chain is difficult to reconstruct.
This is why the governance architecture for AI cannot be built around visible failure events. It has to be built around continuous monitoring of the conditions that predict failure before it becomes visible.
That means tracking whether training data still represents the population the model is serving. It means monitoring output distributions for drift from baseline. It means having a defined owner for each deployed system whose responsibility does not end at launch. It means building the review cycle into the operational budget rather than treating it as an optional overhead cost.
The Real Risk
The conversation about AI risk is mostly about what a model might do at the moment it is deployed. The bias it might encode, the outputs it might generate, the decisions it might influence.
That conversation is worth having. But it is happening at the wrong point in the timeline.
The Air Canada chatbot was not dangerous at launch. It became a liability over time, as the gap between what it said and what the policy actually was went undetected and uncorrected. The Challenger O-rings were not dangerous at the moment they were installed. They became dangerous when the operating conditions moved outside the range in which they had been validated, and nobody in the governance chain caught the discrepancy before it mattered.
Deployment is not the finish line. It is the beginning of the period during which things actually go wrong.
The governance frameworks that treat it otherwise are not governing. They are documenting.
