Frameworks

Automation could ship the wrong order. Decide which outputs get reviewed, and who signs off.

Chris McCarthyNovember 13, 20265 min read
human in the loop aiai agent controlsai order automation riskagentic ai governancesupply chain ai oversight

Automation could ship the wrong order. Decide which outputs get reviewed, and who signs off.

By Chris McCarthy, Founder & Chief AI Officer, OB.1

Once AI starts acting on orders, part numbers or dispatch, a single mistake can repeat at machine speed. The safeguard is a review rule written before go-live that answers one question: what outputs get reviewed, and who signs off? Match the level of review to the cost of being wrong.

This is part six of our Pilot to Production series. This one is for operations and supply chain leads.

From suggestions to actions

The first wave of AI in most companies drafts things. A person reads the draft and decides. The next wave acts. An AI agent can enter a sales order, change a part number on a bill of materials, reroute a shipment or release a purchase order.

That shift changes the risk. A bad draft wastes a few minutes. A bad action ships the wrong part to your largest customer, triggers a contract penalty, or orders a month of material you do not need.

This shift is coming fast. Gartner predicts that by 2028, at least 15% of day-to-day work decisions will be made autonomously through agentic AI, up from none in 2024, and a third of enterprise software applications will include agentic AI. The bar for accuracy stays where it is: by WERC's 2026 benchmarks, top-performing distribution centers hold order-picking accuracy at 99.7% or better. An agent touching your orders has to meet the standard your people already meet.

Why review rules break down

Most companies say they keep "a human in the loop." In practice, that often means a person approves everything, so they stop reading. Or nobody defined what the human checks, so approvals become a click.

Review that adds no judgment is theater. It slows the work and catches little.

Security experts have mapped the risks. The OWASP Top 10 for Agentic Applications, published in December 2025, lists tool misuse and cascading failures among the most critical risks for AI agents. Both describe an agent doing the wrong thing with real permissions, over and over.

The question to ask: what outputs get reviewed, and who signs off?

Sort every action the AI can take by the cost of a mistake, then set the review to match.

Risk levelExample actionsReview rule
LowDraft an order-status replySpot checks on a sample each week
MediumEnter a standard repeat orderAutomatic within set limits, human review outside them
HighChange a part number, release a large PONamed approver signs off before it happens
CriticalChange pricing, ship to a new addressTwo people sign off, with a log

Each row needs a named owner. Write the limits in numbers: order value, quantity change, new customer, new ship-to address.

What a good answer looks like

A review design that holds up in production has five parts:

  1. Every action the AI can take, sorted by risk
  2. Limits written as numbers that trigger human review
  3. Named approvers for each high-risk action, each with a backup
  4. A stop switch that pauses the agent, and a named person allowed to use it
  5. Logs of every action and approval, read by someone every week

How OB.1 fixes this

The Operational Blueprint traces how an order moves through your company today, from intake to shipment. We note every point where a person checks something now, and why. Those checks are the starting point for the AI's review rules.

You get a risk-sorted action list, numeric limits, named approvers and a stop procedure your team can sign. Build partners then put those controls into the system in the Architect phase. Our Govern service runs them after go-live: monitoring, escalation design and quarterly reports on what the agent did and what people caught.

Read the rest of the series

  1. Costs climb faster than your prices
  2. Your ERP data sits in silos nobody trusts
  3. The plant you just bought runs a different system
  4. Staff paste pricing into public AI tools
  5. Your best tech retires, and the fixes leave too
  6. Automation could ship the wrong order (this article)
  7. Teams lean on the system and lose the judgment to step in

Book a Blueprint Session

Sources

Frequently Asked Questions

What does human in the loop mean for AI agents?

A person reviews or approves an AI action before or after it happens, based on how costly a mistake would be. Good designs define what the person checks and when.

Which AI actions need human sign-off?

Actions that are costly or hard to undo, such as changing part numbers, releasing large purchase orders, changing prices or shipping to a new address. Low-risk drafts can use spot checks.

How do you stop an AI agent that is making mistakes?

Build a stop switch that pauses the agent, name who may use it, and keep a log of every action so the team can find and reverse errors.

Who should approve AI actions on orders?

A named person who owns that part of the process, such as the customer service lead for order entry or the purchasing manager for purchase orders, with a named backup.

Get Started

Book a Blueprint Session.

That is the first conversation.

chris@ob1ai.co | (234) 602-0500

Talk to OB.1

Tell us where AI fits in your business. We start with the Operational Blueprint and Diagnostic AI Consulting, and we reply within one business day.