Key takeaways
A coding task needs observable success and failure behavior. “Reject invalid input” leaves open what happens to files already written.
Run the regression test on the original code as well as the patch. In the worked example, detecting an error still allowed a previous export to be overwritten.
Keep requirements, code revisions, commands, outcomes and unresolved limits together. Those artifacts let another session inspect the work without trusting a completion message.
Use persistent specifications and handoffs when decisions must survive a session. A bounded fix can usually proceed from a short task and its tests.
A coding agent can inspect a repository, edit files and run tests within one interaction. Reviewing the resulting change still requires familiar engineering work. The reviewer needs to check that the task describes the intended behavior, that the test exercises the defect, and that the patch preserves behavior callers depend on.
Those checks get harder when the work crosses sessions or tools. A summary may say that validation is fixed while omitting which revision passed, which inputs were checked, or whether a failed operation changed an existing file. The next agent then has to reconstruct decisions from code that may already embody the wrong interpretation.
A small CSV importer makes these distinctions visible. We will follow a written requirement through inspection, a failing command, an incomplete repair, regression tests and a final diff. The example is deliberately constructed. For this article, a Codex assistant wrote the initial implementation, tests and patch and executed them locally. It is a teaching example with real command results, not an independently run agent evaluation, a client case or a measurement of developer productivity.
Define what an invalid import must leave unchanged
The importer reads inventory rows from a CSV file and writes one JSON object per line. Its inputs are a source path and an output path. A successful export should preserve row order, item names and integer quantities, including zero.
The bounded task is to reject missing, noninteger or negative quantities and missing required headers. On a validation failure, the command must exit with an error while leaving an existing export byte-for-byte unchanged. If the destination does not exist, a failed validation must not create it.
That last requirement changes the implementation. Consider this input:
item,quantity
bolts,4
washers,manyThe original program opens the destination for writing, converts each quantity with Python’s int(), and immediately writes that row. It writes the bolts record before discovering that many cannot be converted to an integer. Opening the destination has already truncated the previous export.
The first local reproduction exercised the command as a caller would. It created a previous export, ran the importer against this CSV and inspected both the exit status and the destination. The command exited with status 2 and an error message. The destination contained only the bolts record.
A test that checked only the error would have accepted that outcome. The written task instead requires preservation of the destination on invalid input. The test therefore needs to inspect state after failure, as well as the failure signal.
The contract also states its assumptions. The input is small enough to hold in memory, paths are distinct, and the caller has a writable destination directory. Crash recovery and concurrent writers are outside this task. Without those boundaries, “safe import” could imply a substantially different change.
Inspect the repository before choosing the repair
In an existing repository, the agent first needs the applicable instructions, entry point, callers and test command. It should inspect the current working tree as well. Another person’s edits may explain behavior that differs from the committed version, and must not be overwritten during a repair.
Here, the repository surface is intentionally small. A command-line wrapper parses two paths and calls a conversion function. The conversion uses the standard-library CSV parser, so quoted commas and embedded newlines already have defined handling. The patch keeps that parser. Python’s documentation specifies newline="" when opening CSV files and explains how DictReader maps headers to fields, including missing cells. Python CSV documentation
Inspection identifies both the change and the behavior worth retaining. The defect concerns when the destination is opened relative to validation. Replacing CSV parsing with a string split would expand the patch and introduce a different defect in quoted fields.
The developer, agent runtime, workflow artifacts and execution environment contribute different things. The developer owns the intended behavior and authorizes the scope. The runtime makes file inspection, editing and command execution available to the model, subject to its configured permissions. The written task preserves decisions. The interpreter and filesystem execute the program and expose its effects.
Those contributions depend on one another. A task file cannot enforce output preservation. An executed test can check preservation for its inputs, but cannot decide whether the chosen requirement meets the user’s needs. Birgitta Böckeler’s discussion of coding-agent harnesses makes a related distinction between instructions that guide generation and checks that return feedback after an action. Her account also identifies the limits of relying on generated functional tests. Harness engineering for coding agent users
A row-level check still permits a partial export
A plausible first repair checks that the headers exist, converts each quantity and rejects negative values before writing that row. The companion example retains this candidate as rejected_streaming.py.
The candidate was actually executed against the contract tests. It failed six of ten. Header validation now happened before opening the output, but a bad quantity in a later row still left a partial export. Negative quantities produced an error, yet the prior export had already been replaced.
The candidate confused validating a row before writing that row with validating the import before changing its destination. Both can sound like “validate before writing” in a task summary. Naming the operation and the state that must survive removes that ambiguity.
The corrected implementation collects validated records before opening the destination. Its central change is short:
records = []
with source.open(
encoding="utf-8", newline=""
) as incoming:
reader = csv.DictReader(incoming)
required = {"item", "quantity"}
if not required.issubset(
reader.fieldnames or []
):
raise ValueError(
"required columns: "
"item, quantity"
)
for row in reader:
quantity = int(
row["quantity"]
)
if quantity < 0:
raise ValueError(
"quantity must be "
"nonnegative"
)
records.append({
"item": row["item"],
"quantity": quantity,
})
with destination.open(
"w", encoding="utf-8"
) as outgoing:
for record in records:
line = json.dumps(record)
outgoing.write(line + "\n")The tradeoff is memory proportional to the imported records. That is acceptable under this example’s explicit size assumption. A large-file importer could instead validate into a temporary output and replace the destination after success, with additional decisions about filesystem behavior, cleanup and durability. That alternative was not implemented or tested here.
The final diff adds header and quantity validation and moves destination writing after the input loop. The two positional arguments, JSON-lines representation and CSV reader remain. Reviewing that diff is still necessary even with passing tests. A patch that also changed the output schema or added a dependency would need an explanation beyond the stated defect.
Make the tests detect the original behavior
The tests invoke the command in a subprocess with real CSV files in temporary directories. They inspect exit codes, error messages, destination bytes and decoded JSON records. They do not replace the converter with a mock or merely check that validation code exists.
The same ten tests produced these local outcomes:
Original implementation: seven failures. Invalid imports changed the destination, and negative quantities were accepted.
Streaming validation candidate: six failures. Header checking improved one case, while quantity failures still changed the destination.
Corrected implementation: all ten passed.
Figure 1. The same ten CLI tests distinguish a validation error from preserving the destination. These are executed results from the constructed importer exercise, not a coding-agent productivity comparison.
The positive cases cover ordered rows, a quoted comma, zero, a quoted multiline item and a header-only input that replaces an old export with an empty file. The rejection cases cover a later noninteger value with both existing and absent destinations, a negative value, a decimal, a blank quantity, a short row and a missing required header.
The test inputs matter. A malformed row passed directly to an internal helper might miss how the CSV reader actually represents missing cells. These tests supply CSV text to the real entry point. They exercise the parser and writer together.
The three positive-output tests also failed when temporary copies of the producer were deliberately changed to write wrong quantities or retain an old export on empty input. These limited mutation checks show that those assertions can detect incorrect output. They do not establish exhaustive test coverage.
The two content tests also require decoded quantities to have Python’s int type. Dictionary equality alone would accept 4.0 in place of 4. A producer changed to serialize float quantities failed both revised tests; it passed the earlier value-only checks.
The companion replay reruns the original, rejected and corrected versions and checks the expected outcomes. It retains the complete final diff, raw test logs and interpreter identity. The recorded replay used Python 3.13.5 on Linux; the initial direct reproduction used Python 3.14.4. Neither the failure counts nor the test runner’s elapsed times estimate how coding agents perform on other repositories.
Preserve decisions when the work crosses sessions
For this task, a short contract, regression tests and a diff carry most of the required context. Creating a milestone and several planning documents would add little. The change has one observable failure, one implementation boundary and a focused command to rerun.
Persistent specifications help when several interpretations would lead to different behavior, when a change spans components, or when people must review requirements before implementation. A plan then records dependencies and separately checkable changes. A handoff records the remaining work and the evidence already obtained. These artifacts serve different purposes; repeating the same status across all of them creates opportunities for disagreement.
I maintain coding-agent-toolkit and coding-agent-plugins, which provide workflow instructions and supporting installation components. They are one implementation of this approach. A repository issue, a task file and ordinary Git history can serve the same purposes for a bounded fix.
The toolkit’s handoff contract pairs prose with read-only checks and expected values. Its continuation contract compares current state with those expectations and reports drift. That is documented behavior, not a cross-agent experiment performed for this article.
A concrete handoff for the importer would name the exact revision or file hashes, the interpreter identity, the replay command, its observed results and the remaining output-write limitation. A successor should compare the current files with that revision before relying on the saved result. If the code or tests changed, the earlier pass belongs to the earlier files.
Passing tests leaves a review decision
The corrected importer satisfies the tested validation contract. It still writes directly to the destination after validation. A disk error or interruption during writing could leave partial output. Large inputs could exhaust memory. Concurrent writers, hostile paths and broader CSV schema policies were not tested.
A reviewer must decide whether those exclusions fit the real application. If preserving the last accepted export through a process crash is required, the task and its tests need to cover that failure too. If callers accept untrusted paths, security review has additional work. Maintainability also requires judgment about the surrounding repository, which a synthetic example cannot supply.
The same reasoning applies to agent-generated tests. When one assistant writes the task interpretation, implementation and assertions, it can repeat a mistaken assumption across all three. Running tests on defective code demonstrates sensitivity to the tested defect. It does not independently validate the requirement. Human review, caller examples and existing integration tests can challenge assumptions the new tests share with the patch.
For your next bounded coding-agent task, state the required result and what a failed operation must leave unchanged. Preserve a command that exposes the original behavior, then inspect the corrected diff and the inputs the tests cover. If the work must resume elsewhere, carry those artifacts and the unresolved limits with it. The next reviewer should be able to reproduce the evidence and identify the remaining decision.