Treating tests as disposable artifacts that an AI can write or “auto‑repair” turns a core quality practice into a vanity metric. Tests are executable specifications; they encode intent. When an AI modifies failing tests to match the implementation, the direction of alignment flips—code no longer conforms to the spec; the spec is rewritten to bless whatever the code just did. That erodes trust in the suite.
Overfitting and brittle oracles - Auto‑generated tests tend to assert snapshots and incidental details rather than invariants. You get high coverage with low confidence: lots of lines touched, little behavior protected. The suite becomes noisy, fragile, and hostile to change.
Symptom patching - AI “repairs” often mask root causes—adding sleeps, widening tolerances, or retrying flakiness—rather than forcing the design or concurrency bug to surface. Green builds hide red realities.
Lost design feedback - Hand‑crafted tests (especially before or alongside code) pressure APIs to be cohesive and composable. Offloading that thinking to a generator removes the design conversation that tests are meant to provoke, yielding awkward seams and leaky abstractions.
Metric gaming and review fatigue - A flood of trivial tests inflates coverage and diff size. Reviewers skim or rubber‑stamp, and meaningful failures drown in mechanical churn. The CI signal‑to‑noise ratio collapses.
Maintenance drag - Generated tests often over‑mock internals, freezing accidental structure. Routine refactors trigger wide breakage, encouraging more AI “repairs,” compounding debt and brittleness.
Security and resilience risks - When a boundary test fails (e.g., input validation, permission checks), an auto‑fix may relax the assertion instead of the code. That quietly widens attack surfaces and normalizes undefined behavior.
Knowledge rot - Good tests double as documentation—why a rule exists, what trade‑offs were made. AI‑slop commits rarely capture intent, so institutional memory decays and onboarding slows.
Perverse incentives - If the goal is a green pipeline at any cost, teams learn to tolerate mutable specifications. Quality becomes performative rather than protective.
AI can still help—by suggesting cases, scaffolding fixtures, or highlighting gaps—but it must not be the arbiter of correctness. Keep humans responsible for test intent and treat tests as contracts, not clay. Otherwise, we’ll ship software that “passes” while quietly failing users.
No comments:
Post a Comment