How to Measure AI Automation ROI After Launch, Beyond Hours Saved

A few months after an AI automation goes live, someone in your business has to tell the partners whether it paid off. The easiest number to hand them is the hours-saved total on the builder's dashboard, but that total is an estimate the system makes about itself. The return on investment (ROI) depends on what your business did with the released time and what the automation cost, so the two numbers can be far apart. IBM's 2025 CEO study surveyed 2,000 CEOs in 33 countries and 24 industries between February and April 2025. The CEOs reported that only 25% of AI initiatives had delivered the expected ROI over the last few years. So what gets promised at the start and what gets reported later are often not the same thing.

We think the fix is to decide, before launch, which result the released time has to reach. A landing metric is the one business number, chosen before launch, that the time an automation releases has to move, such as overtime paid, days to deliver the work, or clients taken on without a hire. Below we explain why the dashboard figure misleads, compare three ways to report ROI, and follow one automation three months after launch. Then we cover what the result says about the business case and why the baseline has to come first.

The hours-saved figure is an estimate, not a result

A simple hours-saved counter multiplies how often the system ran by an assumed number of minutes per run. Microsoft's Copilot Studio guidance on measuring agent impact shows how such a figure is built. Its formula for conversational agents counts knowledge references and sessions without them, weighted by outcome, then applies a default of 6 minutes and an hourly rate of $72. In the page's own example, those assumptions turn 10,000 customer-service sessions in a month into an estimated 1,440 hours a month, valued at $103,680 a month, or about $1.24 million a year.

The same page warns against "the time-savings trap": "Claiming value based on theoretical time savings alone undermines credibility." Sessions and user counts, it adds, show usage, not value. So the vendor that publishes the formula also tells you not to rely on it alone.

A counter based only on runs cannot see the minutes people still spend checking and fixing the output, the cases done by hand outside the system, or whether anyone used the released hour for something the business values. It also keeps counting when quality slips. NIST's March 2026 report on monitoring deployed AI systems notes that tests before release happen mostly "in controlled testing environments." Monitoring after launch is needed, it says, to confirm "that AI systems operate reliably as expected in real-world scenarios." A counter that only counts runs reports the same saving whether the output got better or worse.

Three ways to report ROI after launch, and the one that holds up

Once the dashboard figure is in doubt, your team has three common options for what to report instead:

• The counter. It arrives every month without any extra measurement work. It multiplies how often the system ran by an assumed number of minutes, so it shows use rather than value.
• A before-and-after time study. You compare minutes per item before and after launch, using timesheets or a timed sample. It shows that the work got faster, but not whether the business gained anything.
• The landing metric. You read the business number chosen before launch every month against its starting value. It is the only one of the three that ties the released time to a business result, which you can then set against what the automation costs.

Our judgment is that without a landing metric agreed before launch, most teams will end up reporting the counter, because it is the only number that arrives without effort. The time study is a useful middle step, but it stops at the work itself.

Survey data shows why the comparison has to be with the business case rather than with zero. The Wharton Human-AI Research and GBK Collective survey asked about 800 senior decision makers at US companies with 1,000 or more employees and more than $50 million in revenue. The interviews ran from June 26 to July 11, 2025. Of those leaders, 72% reported tracking "structured, business-linked ROI metrics," and nearly three-quarters said they already see positive ROI on generative AI.

Yet the CEOs in IBM's study said only 25% of their AI initiatives delivered the expected return. The two surveys asked different people different questions, so the figures are not comparable success rates. Together, though, they show that a positive return and the return you expected are separate questions, and a project can pass the first and still fall short of its business case.

The answer also depends on who is asked. In the same Wharton and GBK survey, 81% of respondents at vice president level and above believed their return on generative AI was positive, against 69% of mid-managers. Both figures are leaders' own reports from large US companies, not measured results. The gap does not show why the two groups answered differently, but it is a good reason to ask for a baseline and a target that someone else can check.

A worked example: one automation, three months after launch

Here is a hypothetical case that puts the three numbers side by side. An accounting and advisory practice automates the monthly management-report pack for its bookkeeping clients. The system drafts the reports from ledger data, along with notes that explain differences from the budget or the prior month, and an accountant reviews each pack before it goes out.

Before launch, the practice recorded three months of baseline from its timesheets:

• 120 packs a month at 90 minutes each, or 180 hours.
• 60 hours a month of paid overtime in close week, the week the team closes each client's monthly books, at $55 an hour.
• Packs reaching clients on business day 8, on average.

The business case predicted 30 minutes of review per pack, so 120 hours saved a month. It valued those hours at $60 an hour, or $7,200, and subtracted $900 a month in running cost, for a monthly benefit of $6,300. The landing metric, agreed before launch, was monthly spending on extra labor, in two parts: close-week overtime, and a part-time hire planned for this quarter at $3,200 a month to take on new clients.

In month three, the logs and timesheets show the following:

• Volume rose to 130 packs, because the practice took on 10 new clients.
• 120 packs went through the system with 30 minutes of review each, or 60 hours.
• 10 packs had ledgers too irregular for the system and were done by hand at 90 minutes each, or 15 hours.
• 18 of the system's packs needed another 40 minutes of fixing, or 12 hours.
• The owner spent 8 hours on upkeep, such as adjusting the data mapping and the instructions.
• The partners cancelled the planned hire, because the existing team took on the new clients' work.

That adds up to 95 hours of work for 130 packs. At the old pace of 90 minutes a pack, the same 130 packs would have taken 195 hours, so the automation released about 100 hours. The system's counter, which assumes 75 minutes saved per run, shows 150 hours for its 120 runs, while the business case said 120.

In money terms, close-week overtime fell from 60 hours to 15, which saved 45 hours at $55, or $2,475 a month in cash. The work for the 10 new clients, about 15 hours a month at the old pace, went to the existing team, and the $3,200 hire was never made. The hire was planned for new-client work, not to cover close week, so the two savings do not overlap. After the $900 running cost, the practice kept $4,775 a month in money it would otherwise have spent, against $6,300 in the business case.

Turnaround also moved from business day 8 to day 5, a faster delivery that the practice does not count in dollars. Of the 100 released hours, 60 went into the landing metric (45 hours of overtime and 15 hours of new-client work), while 40 reached no measured result. Those 40 hours are reported as unassigned, not as return.

What the measured result says about the business case

The gap between $6,300 and $4,775 is useful, because each part of it traces back to an assumption in the business case. In this example, the review time was right at 30 minutes per pack. What the case missed was the packs done by hand, the fixing time and the upkeep, and it treated every released hour as if it went somewhere. It also valued every released hour at $60, while the measured figure counts only money the practice did not spend. So part of the gap comes from how the hours were valued, not only from how many there were. Those are the assumptions to correct next time, and our guide on how to estimate the value before you build shows where each correction fits.

Suppose the build cost $24,000. On the business case, payback would have taken about 3.8 months, and on the measured figure it takes about 5 months. Both are simple estimates that assume the same saving every month. The first months of a new system often save less while the team settles in, so the partners should track the running total of savings against the build cost rather than divide once. Payback is slower than promised but still within reach, so the more useful decision now is what happens to the 40 unassigned hours.

The partners can give those hours a destination with its own number, such as advisory work with a revenue target. Or they can accept that part of the release stays as slack, which is not counted as return. Our AI strategy work builds "a business case your leaders can fund," and a measured result like this one is how that case gets checked.

Why the baseline has to be recorded before launch

The before number is the hardest one to get later. Once the automation is live, the old way of working usually stops, so its timing is best captured while it still runs. Deloitte's October 2025 report on a survey of 1,854 senior executives in 14 countries across Europe and the Middle East describes the same difficulty in large organizations: "Fragmented systems and siloed platforms make it challenging to track before-and-after impact." The report adds that AI "is typically rolled out alongside broader digital, operational or structural changes," which makes its share of any gain hard to isolate.

For a small practice, the baseline does not need special software: two or three months of timesheet minutes per item, the landing metric's current value and the expected running cost fit in one sheet. It also needs a note of anything else changing at the same time, such as a new hire, because that note helps you check whether something other than the automation moved the landing metric.

Measurement also continues after launch, because the system itself can change. NIST's news release on its monitoring report lists "Detecting performance degradation and drift" among the barriers to monitoring deployed AI systems, meaning it is hard to notice when a system gets worse or starts behaving differently after launch. NIST does not set a schedule for ROI reviews, but that risk is why we recommend reading the landing metric and the hours behind it every month, not checking them once at day 90.

Three recommendations for leaders

1. Record the baseline before you switch the automation on. Take two to three months of minutes per item from timesheets, the landing metric's current value and the expected running cost, and keep them in one sheet the owner controls.
2. Count every hour the automation still needs. From the first day, log review minutes, cases done by hand, fixing time and upkeep. Report released hours only after subtracting them, with the dashboard figure shown next to the measured one.
3. Name the landing metric and its owner before launch. Agree on one business number, a target and a named person who reviews it every month with the partner who approved the spend. Hours that reach no number are reported as unassigned, not as return.

This is how we approach the Operate phase of our agentic systems work, which the page sums up as "Your owner, your runbook, our checks." For ROI, we recommend that those checks include a monthly comparison against the baseline, with the owner answering for the landing metric.

Questions buyers ask

Can we still measure ROI if we didn't record a baseline before launch?

Partly, and the report should say the result is less certain. If the old way of working no longer runs, you have to rebuild the before number from records that already exist. Look for timesheet entries, billing records or email timestamps from before launch, and work out minutes per item from them. If those records are thin, do a small sample of items the old way and time it, then label the baseline as reconstructed.

Should we value the saved hours at our billing rate or our cost rate?

Use the cost rate, unless the released hours produced extra work that was invoiced. A released hour does not earn the billing rate just because it went to client work, since the work may be fixed-fee or would have been billed anyway. Value overtime avoided at what you would have paid, and a hire avoided at its budgeted cost from the date it would have started. If some released hours did produce extra invoiced work, show that revenue on a separate line, minus the cost of delivering it, so the partners can see cost saved and new revenue apart.

How long after launch should we wait before judging the ROI?

Record from the first day, but judge after about three full cycles of the work, such as three month-ends. The first weeks include setup fixes and people learning the new routine, so early figures do not show how the system runs once things settle. Set the review date before launch and put it in the calendar of the partner who approved the spend.

When should we switch an automation off?

Consider it when the running cost and upkeep exceed what the automation saves for two or three review cycles in a row, after you have tried to fix it. Another reason is a landing metric that has not moved even though plenty of work goes through the system. If very little work goes through it at all, check first whether staff have gone back to the old way, because that has its own causes and fixes, covered in our playbook on what to do when staff stop using AI after rollout. Before you switch it off, record the measured figures and the reason, because tools change quickly and a task that did not pay off this year may be worth another look later.

After launch, judge an AI automation by how far it moved the business number you chose before launch, measured against your own baseline, rather than by the hours it counts for itself. If you want a system built with a named owner, a runbook and scheduled review points from the start, see our agentic systems work.