Measuring AI agent ROI: cost, quality and time KPIs with formulas

Where do you start measuring AI agent ROI?

To measure an AI agent's ROI, compare the same process before and after the agent on four groups of metrics: cost per completed task, output quality (against a human-reviewed sample), cycle time and risk. ROI only means something when you have a baseline measured before the pilot and when costs include the human time spent reviewing, fixing and handling exceptions.

This is not a theoretical concern. Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. The last two are measurement problems: without measurement you can neither prove value nor see costs creeping up.

This guide gives a measurement framework that does not rely on anyone else's benchmark. Every number in the examples is illustrative; real numbers must come from your own process. If you have not mapped the process yet, start with how to map an AI workflow.

Step 1: Take a baseline before the pilot

A baseline is a snapshot of the current, human-run process over a representative period. Without it, any claim like "we saved 60% of the time" is a feeling. Pick a narrow process with clear inputs and outputs, such as triaging support emails or matching invoices to purchase orders.

NIST's AI RMF 1.0 makes the same point: risk management for AI systems meant to augment or replace human activity requires some form of baseline metrics for comparison, which is hard to systematise because AI performs tasks differently from people. So measure the baseline at the level of task outcomes, not individual steps.

• Write down what one task is and what done means: when it starts, when it ends, what counts as acceptable.
• Record volume: tasks per day or week, broken down by type and difficulty.
• Measure actual touch time and wait time per task on a sample rather than estimating from memory.
• Measure current quality: error rate, rework rate, related complaints. People make mistakes too, and the baseline must reflect that.
• Record current cost: hours worked at a fully loaded hourly cost (salary, benefits, management) plus software already in use.
• Fix the measurement window and keep the same definitions for the after period.

Step 2: Compute cost per completed task

The key money metric is not the monthly API bill but the cost per task completed to standard. A low bill means little if people still fix most of the output.

• Formula: Cost per task = (Model cost + Amortised infrastructure + Review hours × hourly cost + Exception hours × hourly cost) ÷ Tasks completed to standard.

Step 3: Measure quality on a reviewed sample

Quality is not measured by a feeling that output "looks fine". The reliable approach is to draw a random sample of outputs, have a qualified person score them against written criteria, and compare with the same scoring applied to the baseline. The NIST AI Risk Management Framework (AI RMF 1.0) likewise stresses that AI systems should be tested before deployment and regularly while in operation.

• Sampled accuracy: outputs meeting criteria ÷ outputs scored. Record sample size and sampling method.
• Rework rate: tasks corrected or redone after delivery ÷ total tasks.
• Escalation rate: tasks the agent hands to a person ÷ total tasks. This is not bad in itself; an agent that escalates at the right time is behaving well.
• Missed escalations: of the tasks the agent handled alone, how many should have been escalated? This is the more dangerous failure.
• Critical errors: count consequential errors (wrong information sent to a customer, wrong figures in accounting) separately from minor ones.

Step 4: Measure cycle time and wait time

An AI agent often does its part very quickly, but the customer or receiving team only cares about total time from request to completion. If the task still sits waiting for review for half a day, cycle time barely changes.

Split cycle time (from when a task arrives to when it is completed) into processing time and wait time. Most of the improvement often comes from reducing waiting, for example by scheduling review windows or letting the agent close agreed low-risk cases on its own. Report the median and a high percentile (for example the 90th), not just the average, because a few stuck tasks can hide inside an average.

Step 5: Track risk metrics alongside

Effectiveness is not only faster and cheaper. NIST AI RMF describes the Measure function as using quantitative, qualitative or mixed-method tools to analyse, assess, benchmark and monitor AI risk and related impacts, including tracking metrics for trustworthy characteristics and human-AI configurations. For a business AI agent, you can start with these simple metrics.

• Incidents per month by severity, plus time to detect and time to resolve.
• Actions blocked by permission or safety rules (a sign the agent tried to act outside its scope).
• Share of tasks with complete trace logs: input, tools called, who approved.
• Cases where personal data went somewhere it should not.
• Stability over time: this week's sampled accuracy versus previous weeks, to catch degradation when data or models change.

AI agent KPI table with formulas

The table below collects the main metrics. Pick the 5–8 that fit your process instead of tracking all of them.

How to run a fair before/after comparison

Illustrative example: a customer service team measures a 4-week baseline where each request takes 12 minutes of work on average. In the pilot, the agent drafts replies and a person reviews them in 4 minutes, but 15% of requests must be rewritten from scratch. Measured properly, the real time saved must subtract the rewrites and the model cost; it is not simply 12 − 4. These numbers only illustrate the method.

• Same task types and difficulty: if the pilot only receives easy cases, results will look artificially good.
• Same quality criteria and, where possible, scorers who do not know whether output came from the agent or a person.
• Long enough to get past the learning period and include normal volume swings.
• Where possible, run an agent group and a non-agent group in the same period to remove seasonal effects.
• Set success and stop thresholds in writing before looking at results.
• Log every change during the pilot (prompt, model, process) so you know what caused the result.

Common mistakes when measuring AI ROI

The seven mistakes below make ROI look good in a report but fall apart when you scale.

• No baseline, then comparing against a manager's estimate.
• Counting only the API bill and ignoring review, correction and exception time.
• Counting tasks the agent touched instead of tasks completed to standard.
• Converting every saved hour into money when those hours are not redeployed to valuable work.
• Using averages that hide stuck tasks or rare critical errors.
• Adopting a vendor benchmark or industry survey as the expectation for your own process.
• Measuring once during the pilot and stopping; quality and cost can shift in production. See from AI pilot to production.