Key takeaways
A success score can overlook required procedure or permission, or reject a valid result. Establish what completion means for the actual service.
Validate the checks against independently assessed cases, including outputs accepted and withheld. Detecting planted errors does not establish performance on routine work.
Follow flags, retries and interrupted writes through to their outcomes. Detecting a defect does not establish that the workflow prevents or resolves it.
Count every eligible request, including justified no-action decisions and unfinished work. Report correct completion, human effort and waiting time alongside the model’s score.
Introduction
A high benchmark score does not establish that an AI-assisted service is ready for routine use. A system can supply correct fields while skipping a required approval. It can flag an error but leave the request unfinished, or deliver the faulty result anyway. An evaluator can also reject valid work. Each failure calls for a different repair, and a single success rate can conceal them.
For the person deciding whether to deploy the workflow, the question is which work it can be allowed to complete, under what conditions, and with how much human involvement. This article explains how to build the evidence for that decision. It examines what success scores recognize, whether checks detect the errors that matter, what happens after a failed attempt, and how to account for work still waiting on people.
Consider a hypothetical support service receiving scanned address-change forms through authenticated customer accounts. The forms use varied layouts and may contain handwritten corrections. The AI component reads the document and proposes the shipping-address fields. Ordinary application code supplies the account identity, enforces permissions and writes accepted changes. A structured online form could eliminate the document-reading step; the evaluation concerns the document intake that remains, not a claim that address changes inherently need AI.
The service can apply an authorized update and confirm it, close an already satisfied request without a change, or ask for missing information. The example’s rules and later counts are invented. They let us follow output correctness, permission and completion through one fixed task.
Correct fields do not prove the request was completed properly
Preparing a recommendation and completing a record change require different evidence. A recommendation can be complete when it identifies the customer, supplies the proposed address, and points to the supporting document. If a requested change is needed, the request remains unfinished until an authorized actor applies it and confirms the resulting record. An already satisfied request can close without a write when the workflow verifies the current record and the policy permits that outcome.
For the hypothetical update service, substantive correctness means that the intended customer’s shipping address agrees with the submitted request. Acceptable formatting may vary. An address with an expanded street abbreviation can be correct even if it differs from a stored reference string. An address that omits an apartment number can be wrong while satisfying every schema rule.
Required procedure concerns how the work happens. Suppose this service requires the workflow to check the current record before writing, retain the source document, and read the record after the update. A final address match cannot establish that the workflow performed those steps. The evaluation needs the corresponding observations and action records.
Authority limits which changes the workflow may make. Here, the customer’s authenticated submission approves the requested shipping-address change. The service owner separately permits automatic execution for unambiguous account matches; an authorized staff reviewer must resolve and approve exceptions before a write. Staff approval must stay within the customer’s request. Changing a billing address or another customer’s record falls outside that permission. A technically successful write outside that permission fails the acceptance conditions.
These distinctions apply to assistants, fixed workflows, and agents. An assistant may stop at a recommendation. A fixed workflow follows predefined steps. An agent may choose tools or ask follow-up questions. Each needs evaluation at the point where its work will be used; systems that act also need evidence about what they changed and whether they were authorized.
The owner should decide which requirements each outcome must satisfy and which can be managed across the service, such as waiting time or review capacity. In this example, good field accuracy cannot compensate for an unauthorized write.
Success labels can overlook violations or reject valid work
An evaluation can label work successful while omitting a requirement the owner needs. The Beyond Task Completion preprint examines simulated support trajectories that reach the benchmark’s goal but receive procedural-violation flags. Its manual airline audit covers all 131 flagged successes. Confirmation rates ranged from 93.8% to 95.2% across the three agent models. Auditing only flags cannot estimate violations missed among unflagged runs.
For the address workflow, a grader that compares only the final shipping-address field would miss a required approval step. Improving that grader means supplying the approval evidence and checking the requirement. Averaging an authority failure with several correct fields would still violate this example’s acceptance rule.
An evaluation can also reject valid work. AgentRewardBench describes a web agent that correctly names Acadia National Park in an explanatory answer. The rule grader rejects it because the answer differs from the configured exact string. In Table 3’s comparison across WebArena, VisualWebArena and WorkArena++, rule-based estimates were at or below expert success rates. The GPT-4o judge generally overestimated success, with two exceptions in WorkArena++.
Inspect the task instructions, grader inputs, reference labels, and acceptance rule before interpreting the score. A missing requirement calls for a broader evaluation. A false rejection calls for recognizing valid alternatives.
Detecting a defect does not establish that delivery was prevented
Reliable automated checks can reduce evaluation cost. Choose checks that can establish the required property from available evidence, accounting for their implementation, operation and review costs. A schema validator can check that an address has required fields. An authorization service can check a request’s permission against the intended write. A comparison with the resulting record can check what changed.
Checking permission, enforcing it, and auditing the result are separate jobs. In this example, the record-management API must enforce the owner’s execution scope and refuse writes without evidence linking the actual change to the authenticated customer’s request and any required staff approval. The producer must have no write path that bypasses this enforcement. Test attempts with missing approval, a wrong account, or an unauthorized field, and inspect the resulting records even when the API reports refusal.
Begin checker, permission and interruption tests offline, using replayed requests and copied records. A shadow run can assess proposals alongside the incumbent process while staff control the writes. A bounded live pilot under the owner’s interim authorization can then establish how automatic execution, review queues and staff response behave together on current intake.
A checker that sees only extracted fields cannot establish whether the extraction omitted text from the original document. A checker that reads only the producer’s explanation cannot establish whether the record changed. Giving a model a longer prompt does not supply either missing observation. Specify what each checker sees and which defect classes that evidence can reveal.
Validate the checker on accepted and withheld candidates
The evaluation also needs reference judgments established independently of the checker’s labels. For the hypothetical workflow, reviewers could inspect the original form, authenticated request, approval record, and before-and-after account state against written criteria. Reviewers should assess candidates before seeing the checker’s verdict where practical. Record accepted address equivalences, adjudicate disagreements with reasons, and preserve unresolved judgments.
Sample naturally occurring candidates across the intake, including those cleared for delivery and those withheld, both flagged and unflagged. Inspecting only flags estimates how often flags are justified and overlooks escaped defects. Inspecting only cleared candidates overlooks false rejection and its workload consequences. If difficult cases are oversampled, retain the sampling probabilities so estimates for the incoming population can be weighted appropriately. Keep initial and repaired candidates identifiable, and state which population each error rate describes. Keep a held-out sample that was not used to tune the checker or its prompt.
With a material defect defined as the positive class, sensitivity is the fraction of defective candidates the checker flags. Specificity is the fraction of acceptable candidates it leaves unflagged. Report both by the defect classes that affect the decision, with sample counts and uncertainty. If a sample contains no defects of a class, it does not estimate sensitivity for that class. Agreement with a success label is not automatically sensitivity to a consequential defect.
Deliberately planted challenges answer a different question. An address with a removed apartment number can test whether a checker detects that omission. The challenge should reproduce an error the actual extraction pipeline can produce, and the checker should see the evidence it would receive in operation. Such tests can examine rare defects that natural samples contain too few of to estimate sensitivity. Keep challenge results separate from natural-error estimates. Detecting every planted omission does not establish how often omissions occur or how often other defects escape.
Published evaluator validation can guide this work, provided its measurement matches the intended use. RAGChecker diagnoses retrieval and generation in retrieval-augmented generation systems. Its meta-evaluation compares metric differences with human preferences on 280 response pairs; annotators also see GPT-4 critiques. Those critiques can influence the reference judgments, which is one reason to document what reviewers see. The study validates response comparison; catching material defects before delivery requires a different measurement.
ARES demonstrates another use of automated judging. It combines specialized judges with human-labeled data to correct aggregate estimates and construct confidence intervals for RAG system assessment. The paper reports useful transfer across some query and document shifts. Its judges did not generalize to the tested larger shifts involving language, code, or extraction. The result supports cheaper aggregate measurement under conditions where the judges and reference labels have been validated.
Follow a flag through to the final result
A detected defect prevents delivery only if the delivery policy acts on it. Suppose a checker flags a proposed update to the wrong account before any write. The workflow withholds the update, resolves the account match, obtains staff approval within the customer’s request, and checks the repaired candidate. It then applies the change and reads the target record. Only that confirmed outcome completes the request. The initial flag, repair attempt, and checking effort remain in the evaluation record.
If the workflow merely logs the warning and applies the wrong update, the checker detected the defect but the delivery policy failed. If it withholds the update indefinitely, it prevented that write but left a service request unresolved. Evaluate the final result, including any new defect introduced during repair. A flag count alone measures neither outcome.
Figure 1. Three possible outcomes after a checker flags a hypothetical address update. Confirmed repair, an unfinished request and faulty delivery require different accounting; detecting the defect alone does not establish completion.
Repeated attempts and interrupted writes need separate recovery tests
Repeated attempts can improve the chance of producing a satisfactory candidate. They also create selection work. A system that generates several addresses needs a rule for choosing among them, and that rule can choose the wrong one even when a correct candidate exists.
The original tau-bench paper adopts pass@k from code generation to measure the chance that at least one of k independent trials of a task succeeds. It introduces pass^k for the chance that all k trials succeed, averaged across tasks. Both concern repeated same-task trials. A production selector or recovery procedure needs its own evaluation.
For the address workflow, test the actual attempt limit, selection rule, and escalation route. Compare the final delivered results with a single-attempt policy under the same intake and resource constraints. Include failed attempts in the cost and elapsed time. Retrying an extraction that repeatedly loses the apartment number may reproduce the same mistake. A second checker reading the same incomplete extraction may miss it too. Measure joint misses and what the additional check catches; different model names do not establish independent errors.
Task difficulty also affects the interpretation. Suppose half the tasks have a 90% single-trial success probability and half have 10%. Repeating one randomly selected task twice gives a 41% chance of two successes: (0.9² + 0.1²) / 2. Drawing two independent tasks instead gives 25%: ((0.9 + 0.1) / 2)². The first experiment measures repeated performance on the same task; the second concerns independently arriving requests. In general the corresponding quantities are E_t[p(t)^k] and E_t[p(t)]^k, under the stated independence assumptions.
A service forecast therefore needs evidence about independence and the stability of the incoming mix. A shared outage can correlate outcomes; changing arrivals can alter the marginal success probability. Test those conditions directly before using a repeated-trial score as an operating forecast.
Recovery also changes when a system writes state. Suppose the update succeeds but its confirmation times out. A retry must first establish what happened in the record system. Record the requested change and reconcile externally recorded state before issuing another write. Test interruption before the write, after the write, and during confirmation using the configured recovery policy. If an unauthorized change happened before repair, retain the incident even when the final address is correct.
Include no-action decisions and unfinished requests in completion rates
Accurate results on easy requests can coexist with an unfinished review queue. Define the eligible incoming population before running the evaluation, including cases that later require review or additional information. Record out-of-scope arrivals separately with the reason for exclusion. Do not remove an eligible case from the denominator because the workflow refused it or exhausted its attempt budget.
Choose an intake window and a follow-up horizon. For illustration, suppose the owner authorizes a bounded pilot after the offline checks. The address service receives 100 eligible requests during a week and follows each for two business days after arrival. The following invented counts demonstrate accounting only. They are not pilot results or proposed performance targets.
For this ledger, any staff intervention needed to complete a change, including review, repair or approval, counts as human fallback even if software performs the write. The automated route requires no such intervention before completion.
Table 1. Invented counts illustrate request accounting, not performance. The 90 reported closures need independent auditing; ten requests remain unfinished.
The delivery policy reports 90 closed requests: 82 completed-change requests and eight no-action closures. Reported closure coverage is 90/100; independently validated success remains unknown. Audit both changes and no-action closures against the acceptance conditions. A conflicting request wrongly closed as a duplicate is an escaped failure. Keep audited defects as attributes of the cases, including any violations during requests that were later repaired or remain unfinished.
A request that is flagged, repaired, reviewed, and completed occupies one final-disposition row. Its intermediate events remain available for diagnosing failure and measuring effort. A handoff that finishes a change by the horizon counts as human fallback. A handoff that is still waiting counts as unfinished. Report late resolutions when follow-up continues, while preserving the original deadline result.
Closing without a change needs evidence too. If the requested address is already present and the policy permits closure, no action can be correct. If the form conflicts with the current record and the workflow refuses to investigate, the request remains unresolved even if the policy marks it closed. The FixedBench study tests coding agents on already resolved and partially repaired issues. Instructions encouraging abstention reduce unnecessary edits, but also increase abstention on partially repaired issues that still need work. That tradeoff supports separate labels for correct no-action and unmet obligations.
Report quality among the 90 reported closures and successful-completion coverage over all 100 eligible requests. With a complete audit, divide closures meeting the acceptance conditions by 90 for closure quality and by 100 for successful-completion coverage. For escaped defects in reported closures, divide defective closures, including wrongful no-action, by 90. Dividing that same numerator by 100 reports the share of eligible intake with a defective closure. A defect rate restricted to the 82 completed-change requests counts affected requests among those 82; it does not measure defects per individual write. The ledger does not count write events. Report the ten pending requests separately.
Also report the share of all 100 eligible requests affected by any harmful or unauthorized action, including unfinished requests and those later repaired. Count each affected request once in that measure, even if several actions occurred. A correct final address does not erase an earlier unauthorized change. Report action-event counts separately if needed; a rate per action requires an inventory of the relevant action events, including actions during unfinished and repaired requests. Neither 90 closures nor 82 completed-change requests supplies that denominator.
These formulas use counts established by a complete audit. If only a sample was audited, estimate the relevant population counts or proportions under the audit’s sampling design, with weighting where needed and uncertainty. Do not divide raw defects found in the sample by 90, 82 or 100 and present the result as the corresponding population defect rate. Report the sampling method and sample counts, and leave unaudited outcomes unverified.
Ten completed-change requests used human fallback, but that count is not total human involvement. Closures and unfinished requests may also consume staff time. Measure operational labor across all routes, including review, corrections and requests for missing information; report evaluation and audit labor separately. Record customer effort supplying clarifications separately from staff labor. Measure active labor and waiting time separately, plus the queue remaining at the horizon. Compare the configured workflow with the incumbent process on comparable requests, including the incumbent’s errors and unresolved work under the same resource budget.
Use the evaluation to set an operating scope
The evaluation should support a specific operating decision. The invented ledger supports examining the ten pending requests and the demand for human fallback. Its counts alone cannot justify wider permission to write. That decision also needs independent outcome audits, authorization-enforcement tests, recovery evidence, and measured labor and waiting times.
For the hypothetical service, the owner might permit routine automatic shipping-address changes within the tested class of authenticated requests with an unambiguous account match and the required approval evidence. Conflicting or incomplete requests would take the defined review or information-request route. That scope is defensible only if the evaluation supports its acceptance conditions and the service can complete the resulting human work within the agreed time and capacity.
Set the decision criteria before inspecting the final results. They should cover escaped defects by consequence, completed work, appropriate no-action outcomes, unresolved requests, waiting time, and human effort. The owner chooses thresholds from the actual service obligation and tolerable consequences. A favorable average score cannot override a requirement designated mandatory for each resolved request.
Scale the evaluation effort to the consequences and reversibility of mistakes. A narrow rollout with monitoring, rollback and post-delivery audits may be adequate for readily reversible, low-consequence actions. Actions whose consequences cannot be undone need stronger evidence before delivery. In either setting, count every eligible request, enforce authorization for writes, and follow flags through to final outcomes. Scale sample sizes, independent adjudication, planted challenges, interruption scenarios and release-record detail to the failures the owner must be able to detect and the uncertainty the decision can tolerate.
Preserve a short release record with the eligible population, input period, follow-up horizon, and comparator. Identify the producer, parser, tools, checker, prompts, permissions, and delivery-policy versions. Record the final outcomes, reference-label method, checker findings, resource limits, and uncertainty supporting the decision. Include exclusions and evidence that was too sparse to support a conclusion. Keep that record with the configuration whose use it supports.
Sparse evidence can justify a narrower scope. If the evaluation supports recommendations but cannot establish authorized execution and reliable recovery, retain the recommendation boundary. If review demand exceeds capacity, revise the routing policy or restrict the service volume and evaluate the revised process. Report those limits as part of the decision.
Name what would require re-evaluation. A changed document format can invalidate extraction evidence. New permissions can change which actions are acceptable. A checker or model update can change which defects escape, and growing review delays can make the service miss its obligations. Preserve cases that reveal those changes and recheck the affected acceptance conditions before expanding the supported claim.
Further reading
For the broader evaluation framework, see Evaluating AI-assisted systems. For how evaluation fits into discovery, design and delivery, see Implementing AI in business workflows.