Teams lean on the system and lose the judgment to step in. Decide who owns adopting new model capabilities.
By Chris McCarthy, Founder & Chief AI Officer, OB.1
As AI takes on more routine work, fewer people practice the judgment needed when a sensor fails or an edge case appears. Every model upgrade moves that line again. The fix at scale is ownership: who owns the process for evaluating and adopting new model capabilities? Give that job a name, a test set and a schedule.
This is part seven, the last in our Pilot to Production series. This one is for continuous improvement leads and the operations leaders they report to.
The quiet risk of a system that works
The early risk with AI is that it gets things wrong. The later risk is that it gets things right so often that people stop checking. Operators trust the recommendation. Planners accept the schedule. Over months, the skill of catching a bad answer fades because nobody needs to use it.
Research is starting to measure this. A 2025 Microsoft Research study of 319 knowledge workers found that higher confidence in generative AI was associated with less critical thinking. In medicine, a 2025 study in The Lancet Gastroenterology & Hepatology found that experienced doctors' detection rate on procedures done without AI fell from 28.4% to 22.4% after AI assistance was introduced. Aviation learned the same lesson years ago, which is why pilots still train for manual flight.
Then the rare failure arrives: a sensor drifts, a supplier changes a material, a customer sends an order unlike any before. The system gives a confident answer, and fewer people can tell it is wrong.
Why model upgrades make it harder
AI vendors release new model versions often. Each one can be better on average and still behave differently on your edge cases. An upgrade that improves most answers can quietly change the few that matter most to your plant.
Most companies have no process for this. The vendor updates the model, or someone flips a setting, and behavior changes with no test and no owner.
The NIST AI Risk Management Framework treats this as continuing work. Its Measure and Manage functions call for testing and monitoring AI systems across their whole life, well past the first deployment.
The question to ask: who owns the process for evaluating and adopting new model capabilities?
The answer is a named owner with a defined process. In a mid-sized manufacturer, that owner often sits in continuous improvement, because the job looks like any other process change: test, compare, decide, train.
The process needs four parts:
| Part | What it means |
|---|---|
| A test set | Real cases from your plant, including the hard edge cases, with known right answers |
| A comparison | Run the current and new model on the test set before any switch |
| A decision rule | What results justify the change, and who signs off |
| A skills check | Regular drills where people solve cases without the AI |
What a good answer looks like
Treat AI capability like equipment. You would not swap a machine controller without a trial run and operator training. The same rule applies to a model.
Keep people's judgment sharp on purpose. Rotate operators through manual troubleshooting drills. Ask the AI to show its reasons so people can check them. Review the cases where a person overruled the system, and learn from them.
How OB.1 fixes this
The Operational Blueprint identifies where AI is making or shaping decisions today and which of those decisions would hurt most if they went wrong. Those become the edge cases in your test set.
You get a named owner, a test set built from your own operations, a decision rule for upgrades, and a skills plan for the people who need to step in. Our Govern service then runs the evaluation on a schedule, tracks changes across model versions and reports what moved. Our Rules Before Tools training keeps the people side current as the tools change.
Read the rest of the series
- Costs climb faster than your prices
- Your ERP data sits in silos nobody trusts
- The plant you just bought runs a different system
- Staff paste pricing into public AI tools
- Your best tech retires, and the fixes leave too
- Automation could ship the wrong order
- Teams lean on the system and lose the judgment to step in (this article)
Sources
- Lee et al., The Impact of Generative AI on Critical Thinking, Microsoft Research, CHI 2025
- Budzyń et al., The Lancet Gastroenterology & Hepatology, 2025, as reported by News-Medical, August 2025
- FAA, Operational Use of Flight Path Management Systems, September 2013
- NIST, AI Risk Management Framework, AI RMF 1.0, January 2023