Key takeaways
Measure benefits where work is completed and used. Faster extraction or generation does not establish an improvement in the full workflow.
Include setup, checking, correction, fallback and maintenance, including work shifted to other departments. Compare the existing process on the same basis, with its errors and delays.
Saved staff time can create capacity without reducing spending. More output also requires demand and capacity to finish the subsequent work; do not count the same hours twice.
Test improvements in methods, inputs, verification and handoffs against observed errors and delays. Further investment in AI or workflow redesign needs evidence of a worthwhile improvement.
Introduction
A faster task can improve a workflow, but the saving has to survive checking, correction and the work that follows. A document extractor might produce fields sooner while leaving staff with more discrepancies to resolve. A coding assistant might generate a change quickly while increasing the effort needed to get it accepted.
For someone deciding whether to expand a pilot, the unanswered questions are where the saved time goes, what new work the system creates, and whether the organization gets more completed work, lower spending or better service. A tool can shift effort from producing an output to checking it. Faster production can also leave approval queues and downstream delays unchanged.
This article follows a hypothetical invoice workflow to show how to assess those effects. It compares the costs of complete operating plans, distinguishes saved time from cash savings and additional output, and examines what published productivity studies can establish. The aim is to help you decide what evidence would justify an investment, including whether better inputs, checks or handoffs would improve the process more than another model change.
Measure completed invoice processing, not extraction alone
Consider a hypothetical invoice workflow. Staff receive a PDF, extract the supplier, amounts and purchase-order reference, and check the invoice against internal records. They resolve discrepancies and decide how the invoice should be handled.
For a valid payable invoice, processing is complete when it is correctly recorded in the accounting system, approved and included in the appropriate payment run. A documented, justified rejection of a duplicate invoice or invalid charge is another completed disposition. A false rejection of a valid invoice is an error; an invoice still under investigation remains unresolved. Successful, timely payment is a downstream outcome. Inclusion in a run does not establish that payment occurred. Claims about payment benefits must include that subsequent work and its failures.
An AI extractor could shorten the first step. That leaves several possible outcomes. If staff can check its output quickly and the rest of the process can accept more work, invoices may reach payment sooner. If extraction errors require reopening documents and contacting suppliers, correction may consume the saving. If reviewers already have a queue, faster extraction may simply add candidates to it. Each outcome would require a different investment decision.
Checking also involves different questions. Arithmetic can establish that line items sum to a stated total. Matching a purchase order can establish that a reference exists. Neither check establishes that the goods arrived or that a disputed charge should be paid. A plan must specify which properties software can establish, which evidence staff need, and who can authorize the next action.
The whole incoming workload includes invoices that pass checks, require correction, are rejected or remain unresolved. Staff effort and elapsed time answer different questions. An invoice waiting for approval consumes little active effort but can still miss its payment deadline. Follow completed records into actual use; an accurate extraction that never enters the accounting system has delivered only an intermediate result.
The incumbent needs the same scrutiny. Manual entry can introduce errors, reviewers can miss discrepancies, and established approval routes can delay payment. Comparing an AI plan with a faultless manual process would misstate the choice. Record the incumbent’s remaining errors and incomplete work alongside those of each alternative.
Figure 1. Work and cost categories along the valid payable-invoice route in the hypothetical process. Processing on this route ends at correct recording, approval and inclusion in a payment run. Documented justified rejection completes a different route, not shown. A claim about successful payment includes the downstream execution step and its costs and failures. No measured savings are shown.
Software, predictive ML, assistance and delegation need a common comparison
For the invoice example, choose a common budget period that includes implementation and subsequent operation. Use the same expected volume, document mix, deadlines and quality requirements. Include difficult invoices and periods of heavy demand. A plan that handles only a subset needs an operating route for the remainder, with its cost included in the comparison.
A conventional software plan might use document templates, optical character recognition and explicit matching rules. It could suit suppliers whose formats change little. A predictive machine-learning plan might learn to identify fields from labeled examples. Its feasibility depends partly on whether those examples represent the documents it will receive and whether staff can maintain the labels and model as formats change.
Generative assistance could propose fields from more varied documents, with staff checking and acting on the suggestions. Bounded delegation could allow software to record invoices that meet specified conditions and send exceptions to a separate queue. Delegation describes who acts and under what authority; it can use conventional rules, predictive models or generative models. The methods can also be combined within a plan.
Shortlist methods that can meet the requirements, then compare their full cost over the chosen period. A simpler method can cost more if it requires extensive setup or frequent repair. A language model’s ability to extract a field does not establish that it is the appropriate method. Stable formats may make rules cheaper to maintain; changing formats may make those rules expensive. Either conclusion needs evidence from the expected workload.
Feasibility comes before expected returns. A plan needs sufficient review and fallback capacity, access to the required data, and permission to perform its assigned actions. It also needs to respect limits on severe losses. An attractive average return cannot make an unauthorized payment acceptable or fund a rollout the organization cannot afford. Narrowing a plan’s scope may make it feasible, provided the excluded work remains in the cost and outcome comparison.
Use screening estimates to shortlist plans before building them. Start with observed incumbent costs, available prices and estimates of implementation and operating effort. State the assumptions and use ranges where costs or benefits are uncertain. Pilot shortlisted plans when the choice depends on uncertain behavior, such as review effort, exception rates or downstream delays. Scale measurement effort to the commitment, reversibility and consequences of the change, including the downstream work affected. A small trial may be enough for a cheap, reversible change; costly or difficult-to-reverse delegation needs stronger evidence before expansion. Screening estimates help choose what to test. They do not establish realized savings.
Cheap extraction can transfer work to reviewers and other departments
Setup includes preparing data, creating labels where needed, connecting systems, defining acceptance conditions and configuring permissions. Staff need to learn the revised process. During the transition, the organization may run both processes and reconcile their outputs. Count the time spent on this work, including by staff outside the team running the pilot. Past expenditure on the incumbent is already incurred; its future operating and maintenance costs still belong in the comparison.
Recurring costs include software licences or subscriptions, model inference charges and any compute or hosting the organization supplies. Pricing may depend on seats, volume or input length; retries can increase usage charges. Operating effort starts with obtaining usable inputs and includes execution, checking, correction, rejected attempts and fallback. A successful extraction still costs resources if it follows several failed attempts. An abstention may be a sound decision, but the invoice still needs processing. Counting only successful AI attempts would omit both their preceding costs and the work returned to staff.
Acceptance needs a defined meaning. Some records may enter the accounting system automatically, some may pass human review unchanged, and others may enter after correction. Track those routes separately, and check whether accepted records meet the required conditions. Passing the workflow’s checks and being correct are different properties when the checks miss defects. Human review can also miss errors or introduce new ones.
Maintenance continues after launch. Supplier formats change, source records become stale, interfaces change and model updates can alter outputs. Someone must monitor failures, investigate incidents and recheck behavior after changes. Fallback also needs maintenance. A manual route that lacks trained staff or cannot absorb an outage is an incomplete operating plan.
Performance can improve during operation as well as deteriorate. Staff may learn to use the system, or corrections may become reusable training labels if a process actually captures and uses them. Count correction effort once, identifying any reusable investment, and add the separate costs of curation and training. Measure early and later performance on comparable work, recording system versions and staff experience with the tool. Use observed learning and drift to revise the operating forecast; anticipated improvement remains an assumption until measured.
These costs can move between departments. Faster invoice entry might increase the number of exceptions sent to procurement. A reviewer may need information from a warehouse before approving payment. Include that effort and those delays within the outcome being assessed. A pilot’s own dashboard can improve while the organization’s total work increases.
Unresolved work remains part of the measurement period’s result. Report how much arrived, how much met the required outcome and what remains pending, alongside remaining errors and deadlines. A plan that produces cheap accepted records by setting difficult invoices aside has changed its coverage, which must be part of the decision.
Saved staff time is not automatically a spending reduction
Released hours can create usable capacity. If staff spend less time entering invoices while payroll stays the same, they may have time for other work. Small savings scattered across staff and shifts may be difficult to consolidate or redeploy. Cash savings require a change in expenditure, such as lower overtime or contractor spending. Avoided future hiring can also affect the comparison, but it needs a credible account of the demand and staffing the incumbent would otherwise require.
Additional output requires work that someone wants and capacity to finish it. In the invoice example, staff might process a backlog, handle more incoming invoices or resolve discrepancies earlier. More extraction capacity will not produce more processed invoices if approval remains the limiting step. Measure the output after the relevant handoffs, and account for the resources needed there. If demand is limited, the organization may instead use the time for other work or reduce workload pressure.
Keep cash flows and the value of staff time separate. An hourly wage can help describe the scale of effort, but multiplying released hours by that wage does not demonstrate a spending reduction. Valuing redeployed time requires specifying what staff would otherwise have done and what the alternative use produces. Treat the proposed use of that time as an assumption until the organization observes it.
Avoid counting the same benefit twice. Hours used to deliver additional output cannot also be credited in full as labor spending eliminated. A plan can produce both benefits if it allocates different portions of released capacity to each. Record the allocation and the actual spending change, rather than crediting the entire saving to both.
A worked comparison separates capacity from cash
Suppose both plans correctly process the same 1,000 invoices each month, with the same deadlines, justified rejections and unresolved workload. The following figures are invented, not prices or measured savings. Staff hours include checking, correction, fallback and routine upkeep across all affected teams.
Table 1. Hypothetical operating plans for the same monthly work. Hours include checking, correction, fallback and upkeep; costs are invented.
The $6,000 is additional cash expenditure for integration services and temporary staff cover for all internal setup, training and transition work, including parallel running. Assume that cover fully replaces the work displaced during implementation, leaving no additional internal capacity loss to value separately. No imputed staff-time valuation is included in the cash total.
The comparison charges implementation once, then includes twelve full operating months at the stated monthly performance, with stable volume and no discounting. Transition is complete before those months begin; its resources are included in the $6,000. The candidate releases 300 staff hours during operation. If redeployed hours produce $40 each in net incremental value after any additional resources needed for that other work, their assumed value is $12,000. External operation costs $3,600 and implementation costs $6,000, leaving $2,400 of net planning value. Break-even requires at least $32 of usable value per released hour ($9,600 / 300). The hourly value is an assumption about an alternative use, not an observed return.
If payroll stays unchanged and the hours cannot be used productively, this plan adds $9,600 of cash expenditure without the assumed benefit. The same technical result can therefore be attractive as a capacity investment and unattractive as a spending-reduction proposal. A decision needs the organization’s actual staffing and demand conditions.
Quality and service outcomes can justify expenditure even when payroll remains unchanged. Correctly processed invoices, shorter approval delays and fewer unresolved disputes have value. Correct and timely payments are downstream outcomes to track separately. Where a monetary estimate is credible, state how it was derived. Where it is not, report the outcome in its own units and compare the cost of meeting the requirement. A mission objective need not be assigned an invented price to be included in an investment decision.
Apply the same discipline to failures. Count correction effort as operating work. If a corrected invoice still incurs a late-payment charge, that charge is a separate remaining loss. Charging the correction effort again as an identical failure loss would duplicate the cost. Likewise, if an outcome valuation already accounts for defective work, deducting the same defects again would understate its value. Declare the accounting boundaries before calculating a net benefit.
Support and coding studies do not supply a universal AI return
Generative AI at Work, by Erik Brynjolfsson, Danielle Li and Lindsey Raymond, appeared in the Quarterly Journal of Economics in 2025. It studies chat-based technical support for one business-process software company, including workers at third-party providers. The assistant used GPT-3 adapted on support conversations labeled with outcomes such as resolution success and handling time. The study uses data from 5,172 customer-support workers, but the resolutions-per-hour measure covers the subset with call-quality data because some subcontractors did not consistently report those outcomes. The estimated average gain in that measure is approximately 15%. Rollout occurred mainly in fall 2020 and winter 2021. A small randomized pilot preceded the main, observational analysis of the staggered rollout, which compares changes over time as workers gained access.
The assistant suggested responses; workers could edit or ignore them. Less experienced and lower-skilled workers improved both speed and quality. The most experienced and highest-skilled workers saw small speed gains and small quality declines. Published record, author-hosted published paper.
Resolved issues are an operational outcome, so this study establishes more than faster response generation. Its result still does not supply a complete return estimate covering investment and maintained operation. Nor does its average describe every worker or support setting.
METR’s Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, by Joel Becker and colleagues, presents a contrasting result. In its July 25, 2025 revision, 16 developers worked on 246 real tasks in mature repositories they knew well. Tasks were randomized to allow or disallow AI tools from February-June 2025, mainly Cursor with Claude 3.5/3.7 Sonnet. Allowing AI increased completion time by 19%, with a reported 95% confidence interval of 2%-39% longer. On average, developers had forecast a 24% reduction in completion time and afterward estimated a 20% reduction despite the measured slowdown. METR study, version 2.
The outcome includes developers’ reported active work before and after pull-request review, including requested changes. Some post-review times were imputed for unfinished review processes. It excludes other reviewers’ effort and does not measure a complete software-delivery return.
METR’s February 24, 2026 design update adds a further limit. The investigators describe changes in who participated and which tasks developers submitted, alongside lower participation pay and difficulties reporting time when agents ran concurrently. Based on participant conversations, they believe AI likely speeds developers up more in early 2026 than in early 2025, while calling the data very weak evidence for the size of the increase. The update also reports changes in task choice and work quality, so time differences may not represent value differences. It does not establish a reliable current causal productivity rate. Investigators’ update.
The support and developer studies concern different interventions, populations and outcomes. Their percentages should not be averaged into a general AI productivity estimate. Together, they support measuring the work people actually complete in the selected setting. Neither substitutes for observing the costs, remaining errors and downstream outcomes of a maintained operating plan.
Automatic action needs enforced permissions and capacity to handle exceptions
Assistance leaves staff to interpret suggestions and act. That can be an effective use of their judgment, or it can consume time on decisions software could handle adequately. An inexpensive automated check that establishes a required property can reduce recurring effort. Its value depends on what it checks, what happens when it fails and which defects it leaves undetected.
For the same workload and quality requirement, cheaper generation increases the share of total cost spent on checking and correction if those costs change little. Count that effort when judging how far lower generation prices reduce the cost of accepted work. Input quality, implementation costs or downstream capacity may matter more to the next investment.
Delegation changes the operating plan. In the invoice example, software might record a defined class of matched invoices without individual approval of every entry. Payment authority can remain in the existing payment process. The delegated action needs explicit permissions and enforced conditions. Exceptions need a staffed route, and corrections need a defined procedure. Recoverable record entry and an irreversible transfer expose the organization to different losses, even if both are technically possible.
DeepMind’s 2018 account of autonomous data-centre cooling illustrates this design distinction. Operators reported that implementing the earlier system’s recommendations required too much effort and supervision. The later system selected actions using predictions and operating constraints; a local control system checked the proposed actions before implementation. Operators could exit AI control, with fallback to existing rules. The team deliberately narrowed optimization boundaries to prioritize safety and reliability, accepting a tradeoff with possible energy savings. DeepMind’s historical account.
This is a first-party account of non-LLM control, with operators retaining control of the operating boundaries. It illustrates how constraints, automated checks and fallback can support automatic action. It provides no general estimate of generative-agent returns or the net economic gain from delegation.
Compare assistance and delegation on the effort and losses each leaves behind. Individual human approval need not be required where authorization and adequate checks permit automatic action. Conversely, a high average success rate may still leave unacceptable exposure to a rare severe error. Specify limits on individual and cumulative actions, the conditions for stopping, and the capacity to recover or reroute work. Include the resources required to enforce those conditions. Measure the exception and recovery work against the individual approvals removed.
Choose the next investment from measured errors, delays and review effort
Organizational changes can affect what a technology delivers. In The Productivity J-Curve, Brynjolfsson, Daniel Rock and Chad Syverson model how unmeasured complementary investments can cause productivity growth to be underestimated initially, as resources build capabilities whose output is not counted. Growth can be overestimated later, when those capabilities contribute to measured output without being fully counted as inputs. Their historical empirical analysis concerns investments including computer hardware and software. The working paper describes process changes, worker training and other intangible investments. Published record, January 2020 working-paper revision.
For AI workflows, the inference is to examine which complementary work a plan actually needs and when benefits could be realized. In the invoice example, cleaner supplier records could reduce matching exceptions, and clearer approval authority could shorten delays. Those changes may help an AI-assisted process, or they may improve the incumbent enough to change the investment choice. The comparison horizon needs to include the required investment and the period in which its benefits are expected, with that expectation stated explicitly.
Complementary investment does not guarantee that a weak pilot will become profitable. Further spending needs a specific account of what will improve and evidence that the improvement is achievable. Include that spending in the plan. The cost of redesign can exceed the value it makes available, and a technically effective tool can remain a poor choice for the organization.
Choose the next change from observed failures and delays. If extraction errors cause most correction work, compare a better method with better inputs. If accepted records are accurate but checking consumes the saving, test a cheaper check against the properties it must establish. If invoices wait after checks are complete, investigate the handoff or approval process. Verification is one candidate investment among methods, inputs and process design. Its priority depends on the workflow’s measured constraints.
For a pilot, define completion and quality requirements before collecting data. Where feasible, randomly assign comparable incoming work to the incumbent and candidate process during the same period. Shared queues can make one route affect the other; assigning teams or periods may be more appropriate. Where randomization is impractical, compare matched workloads or periods and report differences in case mix, staffing, seasonality and learning. Such comparisons depend on assumptions about those differences and leave more uncertainty about what caused the result.
Use the same completion boundary, cost categories and follow-up period for each route, including rejected attempts and unresolved work. For invoices, link arrival, approval and payment-run timestamps to logs of staff effort and usage charges, including work in other departments. Independently audit a sample of accepted records against the source evidence to estimate defects the normal checks missed. Where rejection is a material route, also audit rejected invoices for justified decisions and false rejections, keeping cases still under investigation separate. Before expanding the plan, check that review, fallback and downstream capacity can support the proposed volume.
Invest further when the measured improvement and a credible account of maintained operation justify it. Revise or stop when they do not. The next decision should name the change being tested, the outcome it should improve and the resources it will require.
Further reading
For the fuller economic framework, see The economics of AI-assisted work. For monitoring costs, outcomes and changes after rollout, see Operating AI systems and deciding when to reevaluate them.