Regression Guard

Problem

You ask the agent to refactor a tax-calculation module — same outputs, cleaner internals. It produces a tidy rewrite, the existing tests pass, you ship it. Three weeks later finance reports that orders shipping to one state are now taxed two cents low. The old code had an undocumented rounding rule for that case. No test covered it because the behavior was load-bearing but invisible. The agent had no way to know it mattered, so it didn't preserve it.

The refactor didn't fail because the agent was careless. It failed because "behaves the same" was never written down anywhere a machine could enforce it. The contract lived only in the old code's execution, and the rewrite replaced the only copy.

This is the structural risk in every agent change to existing code. The agent optimizes for the requested transformation. Behavior the request didn't mention — edge cases, rounding, ordering, error shapes other code quietly depends on — is invisible to it and unprotected by your suite, because suites test what someone thought to test, and nobody thought to test the thing that was just working.

Solution

Before the agent changes anything, capture what the code does now as executable tests, generated from its current outputs. These characterization tests don't assert what is correct — they assert what is current. The change is only allowed to keep them green.

The sequence is strict — the guard goes up before the change:

1. Identify the behavior at risk: the module being changed and
   everything that calls it.
2. Have the agent generate characterization tests by feeding real
   inputs through the CURRENT code and snapshotting the outputs.
3. Run them — they must pass against the unchanged code. A guard
   that doesn't pass before the change protects nothing.
4. Now make the change. The guard must stay green the whole way.
5. A guard that goes red is the decision point, not a nuisance.

The brief for step 2:

Before refactoring lib/tax.ts, build a regression guard.
Enumerate representative inputs INCLUDING boundaries: zero,
negative, the highest bracket, every state with a special rule,
empty and malformed input. Run each through the current
implementation and snapshot the exact output — value, type,
and error shape. These snapshots are the contract. Do not
change the implementation in this step.

When a guard goes red, it is signal — triage it:

The discipline is that changing a snapshot is a deliberate, isolated, explained act — never a reflex to make the bar green again.

Signals

• Refactors that "passed all tests" still broke behavior in production
• Critical paths have load-bearing behavior no test actually asserts
• You hesitate to let the agent touch certain modules at all
• "Make it behave exactly as before" is in the prompt and nowhere executable
• Bugs trace back to edge cases the change silently dropped

Consequences

Benefits:

• The behavior contract is executable, not folklore in someone's head
• Silent regressions become loud failures before they reach production
• You can let the agent refactor aggressively inside a pinned boundary
• The guard outlives the change — it keeps protecting the path afterward
• Forces the real contract to be made explicit, which is its own win

Costs:

• Upfront cost before any value is produced — a tax on every guarded change
• Snapshots that pin implementation details are brittle and breed noise
• Characterizes current behavior including current bugs — it preserves, it doesn't judge
• Worthless unless you defend it: an agent that can edit the guard erases it