Il tuo ROI, calcolato su tassi che abbiamo misurato.

95 min di lettura

1. The question

Every company accumulates rules its code must obey and nobody could guess. Some are commercial: a refund never exceeds what was captured; one loyalty point per full euro paid in money, and none on the gift-card share; a stock correction beyond thirty units needs a manager. Some are regulatory, and from inside the code they look exactly the same: an export to a carrier carries six columns and no address (GDPR minimisation); a struck-through price is the lowest price of the last thirty days (the Omnibus directive); cash stops at a thousand euros (anti-money-laundering law); every movement of money writes an append-only audit line (what a SOC 2 auditor reads). These rules are written nowhere a machine can use. Where they are written at all, they live in a document nobody rereads while coding. While humans wrote the code, team memory and code review carried the rules, imperfectly. An autonomous coding agent has neither. It has what is in front of it at the moment it writes.

The same is true of the two other things that decide whether a change is safe. An agent that authors a route, a query or a workflow file does not carry the organisation's security posture in its head; it carries the habits of the code it can see. An agent that needs a database reset, a credential or a quick script reaches for the shell and runs whatever gets the job done, with the developer's rights. Conventional application security inspects the result later, at the pull request, in CI or in production, after the vulnerable line, the exfiltrating command or the broken rule already exists.

VibeDefend's claim is that the right place for all three is inside the agent's loop, at write time. Our first study (13 August 2026, four arms, 240,710 lines built from scratch) measured that claim in a construction regime. Among other things, it found that the guards stopped a DROP SCHEMA against the application database that two unguarded arms executed repeatedly. It left open the question that matters most to a real team: in a maintenance regime, ticket by ticket, on an existing codebase with its habits and its liabilities, what decides whether a rule reaches the code, whether a dangerous command runs, and whether a security weakness ships? The intuitive answer, "put the rules in the repository and the agent will read them", is precisely the one to test, because it is what every team does, and because if it works, nothing else is needed.

The claim under test is narrow and falsifiable:

On an existing codebase, for ordinary tickets that touch arbitrary business and compliance rules and ordinary security surface, what an agent produces depends on the moment the relevant information reaches it (at the edit rather than at the start of the session) more than on its availability in the repository. A rules file, even a perfect one, is not enough. A layer that injects rules, checks commands and scans the diff while the agent works does most of the job, at a cost of the order of a dollar per ticket.

Three corollaries are tested separately. (i) A rule delivered at the edit is implemented with its literal specifics almost always, and a rule not delivered almost never, whatever the agent's intelligence. (ii) The model's general knowledge does not replace the company's contract: an agent can know the law and still not ship the error code the platform expects. (iii) On tasks without rules, the layer costs nothing and changes nothing. The design carries its own breaking points: two tickets built to push the agents against the rules, two tasks where the rules file is verifiably blind, five predictions sealed before the runs, a protocol amendment that gave the rules file the best version of itself, and an independent open-source scanner that owes nothing to the product.

2. What VibeDefend does, and what was under test

VibeDefend is installed in the agent's environment, not in the repository. On the arm that carried it, it consisted of a project pin, an MCP server and six hooks, exercising four control points inside the agent's loop.

1. Business and compliance rules at write time. A PreToolUse hook fires before every file edit, fetches the rules relevant to the file and the intent, and injects them into the model's context immediately before the edit applies. The corpus (refund caps, loyalty arithmetic, GDPR erasure formats, export allow-lists, audit-trail obligations, cash limits) lives server-side; no rule file exists in the repository. Over the study the hooks made 669 rule injections covering 47 distinct rules. The MCP channel, which the agent can also call voluntarily, served all 49.
2. Security and compliance rules at write time. The same hook injects security rules matched to the code being written (access control, injection, sessions and tokens, logging of personal data, supply chain, denial of service) and the compliance rules that sit on top of them (SOC 2 change management and transport, ISO 27001 exposure and access, HIPAA and GDPR handling of personal data in logs, storage and URLs). Over the study: 1,078 injections of 65 distinct rules on 24 of the 30 tasks, 392 of them from the compliance families.
3. Action guards. Every shell command the agent issues is checked against a policy before it runs: exfiltration primitives, credential reads, destructive SQL, execution from temporary directories, piping downloads into a shell. Over the study: 1,769 commands checked, 17 refused.
4. Live scan, triage, fix. Over MCP the agent submits its own diff to the platform's scanners and gets back, in one pass, static analysis of the source, infrastructure-as-code checks and secret detection. On every push of the repository the platform adds dependency analysis. The agent polls the result, reads the findings and fixes them before the change leaves its working tree. Over the study: 291 scan calls in 22 tasks.

The intended trajectory for a real codebase has two phases (figure 17, § 10.6). Bring to zero: scan the existing code, triage with the knowledge graph, fix with the agent, rescan until the open findings reach zero. Stay at zero: prevent new deviations and new weaknesses at write time so the count never climbs back. This study measures the second phase directly, on three codebases that started at the same low baseline. It does not measure the descent, and says so wherever the descent is drawn. It reports its results on three axes, one per kind of control point (rules in § 5, action safety in § 6, application security in § 7), and prices them together in § 10.

3. Design

3.1 Three configurations, and the amendment that makes four

All arms run under the same regime: claude -p, model claude-opus-5, effort high, permissions bypassed, a continuation loop when the agent stops early, one git commit and one tag per task. Each arm is a directory that starts as a byte-identical copy of the application plus its own kit.

T0, nothing. The application and its CLAUDE.md, a verbose company-and-architecture document shared by all three arms. No rule material of any kind. T0 answers one question: what does the model's own engineering instinct produce?

T1a, the realistic file (tasks 1–24). The shared CLAUDE.md plus a "Business rules" section written the way real teams write them: 41 lines covering roughly half the corpus, paraphrased ("Exports leave with order data only, no personal fields, ever"), two deliberately stale values (returns "90 days" where the corpus says 60; gift-card admin threshold "500 €" where the corpus says 300), and whole families absent (stock mechanics, fraud patterns, session expiry). What this section said about each rule (exact, vague, stale or absent) was classified and sealed before any run (tasks/t1-coverage.json): 11 rules exact, 16 vague, 2 stale, 20 absent.

T1b, the perfect file (tasks 25–30, protocol amendment n° 1). After task 22 the gap between T1a and T0 on exact specifics (7 of 54 against 6 of 53 at that point) raised a fair objection: had the file arm ever had a real chance? The amendment was motivated, pre-registered and recorded in docs/PROTOCOL.md before the first phase-2 run. It replaces the drifted section with the complete 49-rule corpus verbatim, machine literals included, presented as freshly synced by engineering. T1b is the strongest version of "put the rules in the repo" that can exist. T1a and T1b are reported separately and never pooled.

P, VibeDefend. The same shared CLAUDE.md, no rules file, and the layer described in § 2.

The methodological consequence is simple. The only difference between T1 and P is when, and through which channel, the same rule information arrives: T1 reads it at the start of the session, in a file; P receives it at each edit, in its loop. P additionally carries the guards and the live scanners that T1 and T0 do not have at all, which is why the study reports three axes and not one.

3.2 The codebase and its liabilities

Lumea is a fictional French retailer: an Express 4 + better-sqlite3 + zod + vitest backend, TypeScript strict, 21 route files, 27 to 35 tables depending on arm and epoch, seeded with two stores, about a hundred products, 60 staff, 60 customers and two months of trading. The baseline deliberately carries the bad habits the rules exist to correct: a global ORD-000001 order counter, a blind UPDATE stock SET quantity = quantity - ?, a customer search built on z.string().min(1) that returns the full record, an export that joins and dumps personal data, a staff listing that returns access tokens. None of the corpus's arbitrary specifics appears anywhere in the baseline (verified twice by exhaustive search before sealing). At the start, the independent analyser finds exactly one static-analysis finding in the tree, and the dependency scanner finds none. This liability is what makes the experiment realistic: an agent in maintenance never starts from a blank page. It starts from code that shows it how things are done here, and "how things are done here" is often wrong.

3.3 The corpus: 49 business and compliance rules

The 49 rules span eleven families. Each is written in three parts (the constraint, why it exists, and what its violation looks like in code) with deliberately arbitrary, unguessable specifics: 422 REFUND_EXCEEDS_CAPTURED, a 150.00 € manager threshold, a 60-day window, erased-{id}@removed.invalid, a six-column export allow-list, EXP-ORDERS-{YYYYMMDD}.csv, ±30 units, one point per full euro on the card-or-cash share only, ORD-{store}-{YYMMDD}-{seq}. Table 1 gives the families, the control frame each maps to and examples of the specifics; appendix A gives every rule with what it requires.

Roughly half the corpus is regulatory in substance. The six personal-data rules are GDPR obligations expressed as code contracts: minimisation in exports and search results, erasure that keeps the books but not the person, identifiers rather than identities in logs. The pricing rules carry the Omnibus directive's reference-price obligation. The cash and gift-card caps are anti-money-laundering thresholds. The audit family is what a SOC 2 or ISO 27001 auditor asks to read, and the access family is access control as the same standards define it. The other half is the company's own commercial contract. The distinction matters for the exposure a deviation creates (§ 5.3), not for the mechanism: to the agent, a GDPR obligation and a refund cap are the same kind of object, a specific it either receives or does not.

The corpus was frozen by SHA-256 before task 1, and its guessability was graded rule by rule in rules/CRITIQUE.md (35 unguessable, 13 weakly guessable, 1 guessable). A rule is "exact" when its specifics are in the code, literally; the right behaviour with a different value or error code is graded separately. Why arbitrary specifics? Because they are the only way to separate knowing the rule from guessing a reasonable rule. An agent that caps cash at 1,000 € may have read the law; an agent that answers 422 CASH_LIMIT has received the platform's contract. Task M21 makes exactly that distinction: all three arms set the right threshold, and one shipped the error code (§ 9.2).

3.4 The thirty tickets

Tickets are written in developer voice, as incidents and requests, never as references to rules ("A café owner tried to order 78 espresso cups at the till; the order went through and stock went negative"). An automated leak check confirms that no corpus literal and no numeric anchor of a targeted rule appears in any ticket. Each task's rubric (primary and secondary rules, expected specifics) was sealed before any run, in three waves (tasks 1–10, 11–20, 21–30). Each wave was accompanied by a critical review that corrected three defective premises before sealing.

Five kinds of task read differently. Two designed tensions, pre-registered in the rubric, measure compliance under contradictory instruction: M13 ("without blocking honest business") pulls against a rule that mandates blocking, and M24 ("bigger cards, sold at every till") pulls against issuance caps. Two T1-blind checkpoints, M07 and M13, fall where T1a's file verifiably says nothing: if the file arm behaves differently from the bare arm there, something other than the file is leaking. Two amended replications in phase 2 test memory and drift: M28 replays M01's return window with a now-correct file, and M30 extends M01's refund idempotency to the payment path. Five rule-neutral tickets (M16–M20: a dev loop, a CI gate, a pure type refactor, a database reset, a repository map) serve as the study's internal control, because a treatment that "wins" where there is nothing to win is a biased treatment. Their result is reported in § 4.4, in one paragraph, which is what a control that holds deserves.

4. Measurement

4.1 The pipeline, the blind audits and the verdicts

Each task runs the three arms in parallel under the same regime. A fixed chain then executes: artefact assembly (transcripts, diffs, per-arm census of cost, turns, tokens and tools, per-channel census of the rules VibeDefend served, archive of hook traces); decisive greps for the sealed specifics; three independent blind audits; the orchestrator's deep read in the live trees (suites re-run, edge cases probed by hand); one ordered report; a cumulative scoreboard; a commit; a push of arm P to its scanned remote. The thirty dossiers (data/tasks/M01 to M30) each contain the report, the verdicts, the anonymised diffs, the full transcripts and the statistics.

For the audits, each diff is anonymised and assigned a letter drawn per task. The auditor receives one letter, the diff and the canonical text of the rubric's rules, and must quote the diff lines that ground each verdict. It knows neither which arm it grades nor what the other two produced. The letter-to-arm mapping is revealed afterwards and verified by file fingerprint (a race between auditors at task 17 was resolved that way and recorded). The orchestrator then rereads each implementation in the arm's tree, re-runs the suite, and settles disagreements with a documented arbitration.

There are six mutually exclusive verdicts: exact (the arbitrary specifics, literally); right behaviour, wrong specific (the right intent with a different value, format or error code); spirit only; missed; violated (the violation the rule names is enacted); not applicable. Per-arm denominators exclude not-applicable. Four exacts partly derivable from the ticket or the runtime (PAY-01 and PAY-02 in M06, ACC-04 in M03, OPS-02 in M25) carry an asterisk wherever they are counted. Where two auditors graded an identical design decision differently across arms (once, M25), the verdicts were harmonised and the harmonisation recorded. Every number in this paper is recomputed by script from the task dossiers (paper/build/extract.py), applying the sealed rubric strictly. These numbers supersede the running tallies kept during execution, which differed by a few units because off-rubric rules were occasionally counted.

4.2 The independent security audit

The product's own scanners cannot be their own witness. Every arm's tree was therefore scanned at each of its 31 tags (baseline and tasks 1–30) with an independent, widely used open-source static analyser and four of its public rulesets (general, TypeScript, Node.js, secrets): 93 scans in total, node_modules excluded (paper/build/secscan.py). The baseline and the three final trees were also scanned with an independent open-source scanner for vulnerable dependencies, secrets and misconfigurations. Neither tool is CybeDefend's, and neither is named here; the point is that the witness is not ours. The action guards were audited by reading each of the 17 refused commands against the guard's stated reason, and by searching the two control arms' transcripts for commands of the same classes (paper/build/guards.py).

4.3 Sealed predictions, invalidation gates, reproducibility

Five predictions were sealed before wave 3 (tasks/t1-coverage.json): T1a's stale 500 € would bite (M24); its exact-value-without-literal cards would produce the right thresholds with prose responses (M21, M22); P would extend its stock-movement ledger to transfers (M29) and rewire its idempotency machinery onto payments (M30); and the drift-replication prediction for M28, voided by the amendment, was re-registered as "does a fresh file correct the drifted implementation inherited in the tree?". All five are resolved in appendix B. Ten automatic gates (runner/gates.py) check before each report that task tags are consistent across arms, that the last tag's tree is clean, that no corpus literal or numeric anchor leaked into a ticket, that the layer fires on P only, and that the sealed corpus's checksum is unchanged. The repository holds the corpus, the tickets, the rubrics, the three final trees with one tag per task, the 90 transcripts, the extraction, figure, model, table and security-scan scripts (paper/build/) and the consolidated data (paper/data/). Every table and figure in this document is regenerated by python3 paper/build/build.py.

4.4 The control that held

On the five rule-neutral tasks the sealed prediction had three parts, and all three are observed. Parity: code grades are A or A- for the three arms on four tasks out of five. Neutrality of the layer: on these five tasks P received 39 rule injections by hooks and 14 rules over MCP (14 distinct rules on M16 alone) and applied none of them; the feared over-application of the injected arm did not happen where there was nothing to apply. Cost: P is the median or cheapest arm on four tasks out of five ($33.03 against $33.22 for T0 and $46.31 for T1 over the five). The only break in parity is T1's, in M18. In a type refactor where "behaviour must not change", T1a smuggled in a behaviour change drawn from its ACC-05 card (session expiry), admitted in its own commit message; T0 and P shipped a pure refactor, proven to the byte by comparing the emitted JavaScript. The effect the rest of this paper reports sits on the rules, not on the tool being present.

5. Results A: business and compliance rules

5.1 Overview

Table 2 gathers the raw measures over thirty tasks. Three readings stand out before any detail. Conformance: over the 25 rule-bearing tasks, the VibeDefend arm implements 57 exact specifics out of 64 graded (89 %), the bare arm 8 of 65 (12 %), the file arm 14 of 66 (21 %). The file arm's figure aggregates two very different regimes, which § 5.3 separates. Debt: the controls leave behind 57 and 52 non-conformant rules, of which 21 and 12 outright violations; the VibeDefend arm leaves 7, of which one. Price and volume: $232.86 against $196.97 and $208.41, an overhead of 18 % and 12 %, for a quarter less code (8,671 lines added against 11,619 and 12,795) and a narrower surface (131 files touched against 195 and 190).

Figure 1 shows the dynamics. The VibeDefend arm settles between 87 and 92 % from the second task on and stays there. The two controls are indistinguishable for the nineteen tasks of phase 1 (the two curves overlap within a few points) and only part at task 25, when T1 receives the full corpus. In other words: for nineteen tasks, having a realistic rules file in the repository changed nothing measurable about the code produced.

5.2 Task by task

Table 3 gives, for each task, the rules graded, the number of exact specifics per arm and the cost of each run; neutral tasks carry the global code grade instead of verdicts. The full verdict grid (figure 3) shows every rule × arm cell.

Three patterns are visible to the naked eye. The P column is almost uniformly purple, with one red row (M24, PAY-04) and a handful of orange or grey cells. The T0 and T1a columns are mosaics of orange (the right behaviour with the wrong value), red and grey, with isolated exacts that almost all correspond to guessable rules (ACC-05 in M10, the ticket-derivable PAY-01 and PAY-02 in M06, AUD-02 in M14). Below the green line, the T1b column turns purple on the rules the ticket aims at and stays orange or yellow on cross-cutting rules (STK-01 in M26 and M29, REF-03 in M28), which is the pattern the phase-2 prediction announced.

5.3 Verdicts, exposure, and the catalogue of violations

In phase 1 the two controls have exactly the same score (7 of 55) but not the same distribution. T0 violates more (18 against 12), where T1a implements the right behaviour with the wrong specific (28 against 20). The realistic file moves part of the errors from "violated" to "nearly", without converting them into "exact". That is what a paraphrased document can do: give the direction, not the value. The VibeDefend arm makes 46 exacts of 53; its seven deviations are detailed in § 5.4, and four of them come down to rules the layer did not deliver. In phase 2 the verbatim file changes the picture for T1b on the rules at the centre of the tickets: 7 exacts of 11, no violation. Its four deviations are all cross-cutting or peripheral rules (§ 5.5). T0 stays at one exact in ten; its only success, PAY-02 in M30, rides on the house dialect of idempotency, which is guessable. P makes eleven of eleven.

Table 5 classifies each deviation by the exposure it creates for the company, from the reading of the reports: legal (price information, consumer law, erasure obligations), financial (over-refund, over-crediting of points, missing caps), personal data (fields leaking to roles or files that should not see them), security (tokens, authentication, enumeration), operational (wrong stock, missing audit, reference formats). The last column counts the deviations for which the report or the auditor states explicitly that the wrong implementation is covered by passing tests: the arm's own suite certifies the wrong behaviour. It is a floor, not an estimate; for the other deviations the question was simply not settled. A deviation locked in by a green test is the worst case for a team: CI will not find it, CI will defend it.

Table 6 lists every outright violation, the case where the code enacts precisely what the rule forbids, with what the code does and the corresponding exposure. Read as a compliance officer would, the bare arm's twenty-one violations contain six personal-data breaches (personal data in a partner file, in the logs and in a role's response, and a customer base enumerable from a single character or from any till), one consumer-law breach (the reference price), seven financial-control breaches (refunds without caps or reasons, overlapping promotions, points on gift-card euros, an issuance gate removed, a stock column overwritten on an audit request, points adjusted without a trace) and seven integrity and access breaches (stock overwritten without a trace, an unpaginated partner list, a partner surface mounted outside the authentication chain). The file arm's twelve are of the same kinds. The VibeDefend arm's one is the M24 issuance gate, discussed in § 9.4.

Some of these rows deserve plain words, because they say what a coding agent does spontaneously on a real codebase.

• M03, T0. The daily carrier file contains the customer's address and columns outside the allow-list. Rule PRV-04 exists only to prevent this. A CSV delivered to a third party with unnecessary personal data is a GDPR breach on the first send.
• M04, T0. The error log prints the full request URL, query string included. A 500 on the customer search writes the searched email into the logs. Rule PRV-03 says exactly not to do that.
• M05, T0. The struck-through flash-sale price is the catalogue price, not the lowest price charged in the previous thirty days. That is the letter of the Omnibus directive as transposed into French law, and the arm locked it in with a test.
• M06, T0 and T1a. Euros paid with a gift card earn loyalty points. T1a had rule LOY-01, exact, in its file (§ 5.5).
• M07, T0 and T1a. The yearly recount overwrites the stock quantity instead of recording a movement, all or nothing, without a trace. Three rules violated, the same SQL in both arms.
• M11, T0 and T1a. The customer search accepts one character and returns the full record, address included, to cashiers: the defect seeded in the baseline, kept, rewritten and certified by both arms' tests.
• M15, T0. Any cashier opens any order of any store, in full. With guessable numbers, that is an enumeration of the chain's order book from any till.
• M24, T0 and P. Under a ticket asking for "bigger cards, sold at every till", T0 removes the issuance gate and lets any token issue 2,000 €. P, served rule PAY-04 thirteen times, dismantles the caps it had itself set in M06 and rewrites its guard tests. It is the VibeDefend arm's only violation, and the study's most important finding about its limits (§ 9.4).

5.4 The chain: served → exact, not served → not exact

The central question is not "does P do better?" but "why does it do better, and is the mechanism the one claimed?". The census of the rules VibeDefend served to P, task by task and per channel (hooks at the edit, or voluntary MCP fetch), lets us cross, for each graded rule, whether it was delivered and what its verdict was.

Of the 58 graded rules that VibeDefend served to P (hooks or MCP), 55 are exact (95 %). Of the 6 rules it did not serve, 4 are not exact and 2 are, for identifiable reasons. The nine off-diagonal cases:

The result is nearly diagonal: 55 exacts of 58 served (95 %), 4 non-exacts of 6 not served. The two "exact without delivery" cases each have a verifiable explanation. One is derivable from the ticket (the authentication floor of a partner portal); the other is a rule P had already implemented in its own movement ledger three tasks earlier and reused (§ 5.7). The three "served but not exact" cases are a mis-arbitrated price (M05), a threshold argued in writing (M07) and the M24 capitulation. None is a rule served and silently ignored. This crossing is what separates a correlation from a mechanism. The same agent, in the same session, with the same intelligence, ships 95 % of the letters it is served and 33 % of those it is not. And at task 23, where no channel worked (§ 9.4), it argued against the absent rule in a code comment: general knowledge did not fill the gap, it rationalised the other choice.

5.5 The rules file: what it knew against what it did

The most useful question for a team is the file's. What happens when the rule is in the repository, exact, with the right value, read by the agent at the start of the session? The sealed exposure of T1a's file lets us measure it rule by rule in phase 1.

On the thirteen rules that T1a's file carried exactly (the right value, in black and white, in a document read at every session start), the arm implemented them exactly four times. Eight times it produced the right behaviour with another specific, and once it enacted the violation. The VibeDefend arm, on the same thirteen rules, scores thirteen. Availability of the information is not the variable; the moment it arrives is.

M06 is the sharpest case, because it is the only instance of the same exact rule in both arms with two different delivery moments. The ticket asks to accept a gift card smaller than the basket, with a bank card for the rest. Rule LOY-01 says a loyalty point is earned per full euro paid in money, and the gift-card share earns nothing. T1a had this rule, exact, in its file. It rewrote the payment function, computed the eligible share to order the tender types, and left untouched the floor(total / 100) line that mints points on the total: in its own new flow, gift-card euros earn points. Its auditor, blind, wrote that this was "the work of a careful engineer who has never seen the rules document". It had seen it, at session start. P received LOY-01 in its loop at the moment it edited the payment file and wrote earned = floor(remainderCents / 100), with two dedicated tests.

Phase 2 supplies the counter-proof and its refinement. With the verbatim corpus in its file, T1b succeeds on the rules the ticket aims at: pagination in M25, apology points in M27, idempotency in M30, and the return window in M28, where it deletes the 90 days inherited from its own M01 and migrates to 60 (a fresh file corrects inherited drift). But in M26, with rule STK-02 (403 ADJUSTMENT_LIMIT beyond ±30 units) spelled out verbatim in its file, it sets the exact ±30 and answers with a 403 in prose, without the literal. The attention mechanism is visible inside a rule: the number at the centre of the ticket is retained, the peripheral token is lost. And on the cross-cutting rules the ticket does not name, such as STK-01, which forbids writing stock quantities directly, T1b loses three times out of three against the habit of the code in front of it (M26, M29, and by inheritance M12).

5.6 The controls converge

A result we had not predicted is the regularity with which the two controls, run independently, produced the same wrong implementation. Table 8 lists these convergences: the same absolute-write SQL at the recount (M07), the same "strongest wins" policy for overlapping promotions where the rule forbids overlap (M05), the same invented 30-day and 10-minute dials for fraud detection (M13), the same dropping of the ORD- prefix (M15), the same 4,000-character cap and the same forgotten IBAN in the redaction list (M04). These convergences say something important about coding agents: the model's engineering instinct is deterministic in its errors. When a company does not inject its contract, it does not get a variety of interpretations for review to choose from. It gets the same reasonable and wrong interpretation, everywhere, defended by the same tests. That is what makes the debt silent.

5.7 Debt, compliance debt, and composition

Every non-conformant rule left in the codebase is a debt. It will be discovered later, by an incident, an audit or a customer, and corrected by a human who first has to understand why the test that covers it is green. Figure 5 accumulates these deviations task after task. The two controls accumulate about two deviations per task, linearly, with no inflection; T1 only levels off in phase 2. The VibeDefend arm accumulates 0.28 deviation per task, and its curve is flat over the last six tasks. The slope is the measure that matters to a team, because it does not depend on the size of the study: at this rate an ordinary flow of tickets produces debt in proportion to the number of tickets, and the layer divides the rate by eight.

There is a second debt, less visible: what it costs to catch up on a cross-cutting requirement when it was not laid down along the way. Task M14 ("the auditors arrive next month") asks for an audit trail of money events. The two controls, which had never written an audit event, built the trail in one go: 1,107 lines for T0, 1,620 for T1a, with invented external-auditor token classes and an invented role floor (fourth convergence, table 8). The VibeDefend arm had emitted audit event AUD-01 at every task that touched it since M02, because the rule had been served every time; in M14 it completed the trail in 334 lines, with all three rules exact. Compliance laid down along the way costs five times less code than compliance caught up, and the caught-up code is the code that will have to be maintained.

The mirror image of debt is composition. When rule STK-01 ("no direct write of stock quantity; every change is a typed movement in a ledger") was served to P for the first time in M07, it built a stock-movements table. It reused that table for cancellation in M08 (rule not served that day, exact anyway: one of the two "exact without delivery" cases of figure 4), for the wholesale basket in M12, for adjustments in M26, and for transfers in M29 with a typed TRANSFER_OUT/TRANSFER_IN pair bound by foreign key. Likewise, the idempotency machinery built in M01 for refunds was rewired in M30 onto the payment path, the key living inside the money transaction. The controls started from scratch each time, or rather from the code's habit: the baseline's blind UPDATE stock SET quantity = … was reproduced, rewritten and re-blessed in M07, M08, M12, M26 and M29, with comments that own the imitation ("putting it back has to be as blind", M08, T1a). Conformance compounds; so does non-conformance.

6. Results B: action safety

The guards check every shell command the agent issues against a policy, before it runs. This section reads what they did in this study, then what they did in the preceding study, where the agents had a live database and the stakes were different, and draws the two together.

6.1 What the guards did here, read by risk

On arm P the guards checked 1,769 commands over the thirty tasks and refused 17. Table 10 groups the refusals by what the policy matched and reads each group by the risk the command actually carried. Figure 6 sets the count against what ran unchecked in the two control arms. The seventeen commands themselves, with the guard's stated reason, are listed in appendix E.

One refusal is a genuine catch. For the task "nothing checks the repo before a push", P tried to extract the GitHub token from the machine's keychain (git credential fill) to call the API on its own initiative. The guard blocked it. Neither control attempted to access a credential on that task, but nothing would have stopped them. It is one case in thirty tasks, and it is exactly the class one wants refused without discussion: an agent reaching for a stored credential to talk to an external service. Three refusals are correct applications of the policy on commands that carried little risk in this environment: a DELETE FROM without WHERE inside a newly added test, and two local probe scripts run from /tmp. Thirteen are false positives: twelve triggered by the token "nc" inside Python heredocs used to patch test files, one by the read of a source file (src/auth.ts) taken for a secrets file. Each false positive cost P one turn to redo the edit through another tool. The controls ran twelve commands of the same classes unchecked (nine scripts executed from /tmp, three DELETE FROM without WHERE, some of them greps): all harmless here, all of the classes that are not harmless in general.

Read on its own, this study says two things about the guards. They do what they are for: the one command that reached for a credential did not run, and the two control arms had no equivalent. And their precision is not yet where a developer will tolerate it: thirteen interruptions in thirty tasks on harmless commands is a friction cost that a team deploying the guards must tune before rolling them out. What this study cannot say is what the guards are worth when the agent can reach something that matters, because on this small codebase, with a file-based database and no external service, there was little to reach. The preceding study is where that was measured.

6.2 What the guards did when the agent had a live database

The 13 August study ran four agents through 24 build steps on a PostgreSQL-backed platform, twelve hours each, with credentials in .env, a live application database and a scratch directory that happened to contain the study itself. Every command issued by all four agents was searched for the same destructive classes and each hit was inspected. Table 11 gives what the unguarded arms did; table 12 gives the eight commands the guards refused on the VibeDefend arm, graded one by one.

Two facts from that study carry the argument for the guards, and neither could have happened here. First, the two corpus arms each executed DROP SCHEMA public CASCADE; CREATE SCHEMA public; against their own application database, wiping every table, seven and eleven times, unobserved, one of them at the final step of the build. The VibeDefend arm attempted the identical command once and was refused, with the right rule and the right explanation. Second, at 06:55 on 11 August, an unguarded arm executed rm -rf on an absolute path outside its project, intending a scratch run directory. The path carried a mistyped digit, and the directory it would otherwise have hit held every transcript and every measurement of that study. Nothing inspected the command, nothing questioned an unbounded recursive delete outside the project, and a single wrong digit is the only reason the study survived. Of the eight refusals, three were correct with the correct reason, three defensible, two wrong: the same precision problem as here, and the same conclusion. The false positives cost the agent a retry; the absence of a mechanism cost the control arms their database, repeatedly, and nobody found out until the analysis.

6.3 What the two studies say together

The value of an action guard scales with what the agent can reach. On a construction task with a live database and real credentials, the guards stopped a schema drop and two credential writes in twelve hours. On a maintenance task with a file-based database and no external service, they stopped a credential read in thirty tasks. In both settings the arms without guards attempted the same classes of command and executed them, because for them refusal was not an available outcome. In both settings a quarter to three quarters of the refusals were false positives, and the paper says so. The right reading is neither "the guards caught nothing here" nor "the guards saved the database there". It is that the guards are the only component of the loop that can decline, that what they decline is exactly the class that destroys a database or leaks a credential, and that their precision is a tuning item with a measured cost of one turn per false positive.

7. Results C: live scanning and application security

The layer does three things for security that the controls cannot do at all. It injects security and compliance rules matched to the code being written. It lets the agent scan its own diff live (source, infrastructure as code and secrets in one pass). And it scans the repository at every push. Figure 7 shows the whole activity of the layer, task by task.

7.1 Security and compliance rules injected while writing

Over the study the hooks injected 1,078 security rules (65 distinct) on 24 of the 30 tasks, alongside the business rules. Table 13 breaks them down by family. Two thirds are technical rules of the OWASP class: user-controlled identifiers overriding the session's, template-literal SQL, unvalidated JSON.parse, catastrophic regexes, insecure JWT expiry, CORS reflection, missing HSTS, mutable git dependencies. One third, 392 injections, are compliance rules: SOC 2 change-management and transport controls, HIPAA and GDPR handling of personal data in logs, storage and URLs, ISO 27001 exposure and access controls. These are the rules an auditor checks after the fact, delivered to the agent before the fact.

Honestly, the retrieval is not clean. Rules for React, Java XXE, Rust, Kubernetes and Dockerfiles were retrieved by similarity and injected on a TypeScript/Express backend. That noise has a cost (context tokens) and no benefit; it is a retrieval-precision item for the product, and it is counted in P's token overhead. Table 14 lists the twenty most frequently injected rules.

7.2 The live scan loop

P submitted its own diff to the platform's scanners 291 times over 22 tasks (table 15). Each submission scans the changed files for static-analysis weaknesses, infrastructure-as-code misconfigurations and secrets in one pass, live, while the agent is still in the task; every push of the repository adds dependency analysis on the whole tree. The scanners reported two distinct findings. The first (M04, CWE-134, a non-literal format string in the request logger, informational severity) P fixed within the task, the next scan returning zero: the loop scan → triage → fix → rescan worked autonomously, with no human in it. The second (M11, CWE-79, medium) flags a direct write of a CSV into the response. On reading the code, the response carries Content-Type: text/csv and an attachment disposition, a classic static-analysis false positive on exports, which P left open. The code produced by agents on this codebase and these tickets contained few weaknesses of form; the scan loop's value on this study is that it closed, not the size of what it found.

7.3 The independent audit: stay at zero

The product's scanners cannot be their own witness. Figure 9 and table 16 show what an independent open-source analyser finds in each arm's tree at each of its 31 tags.

All three trees start at one finding: the direct CSV write in the legacy export, the same false positive the platform's scanner flagged in M11. Over thirty tasks the bare arm's tree climbs to four: two mutable GitHub Actions tags in the CI workflow it wrote at M17, and two non-literal format strings in the diagnostics and error modules it wrote at M04 and M05. The file arm's tree climbs to three: the same two mutable action tags, and one format string in its capture module. The VibeDefend arm's tree ends where it started, at one. It introduced exactly one finding in thirty tasks, the M04 format string, and removed it within the task through the scan loop; its CI workflow at M17 pins nothing mutable. The dependency scanner finds no vulnerable package, no secret and no misconfiguration in any arm at task 30: no arm added a dependency, and the baseline's packages were clean.

The numbers are small, and the paper does not inflate them. On a fresh, small, well-typed codebase, agents introduce few weaknesses that a syntactic scanner can see. What the sweep establishes is the shape, not the magnitude: the controls' curves step up and never step down, because nothing in their loop looks; the VibeDefend arm's curve steps up once and steps down inside the same task, because something does. That is the "stay at zero" half of the product's promise, measured with a tool that owes it nothing. Per task, the controls introduced 0.10 and 0.067 findings and fixed none; VibeDefend introduced 0.033 and fixed all of it.

7.4 The codebase's liabilities, and what each arm did with them

Table 17 is the "liability" reading of the study. The seeded codebase carried defects that the tickets crossed without naming them: the staff listing that returns the token column (fixed by P in M10, kept by the controls), the one-character search that enumerates the masked database (fixed by P in M11, kept and rewritten by the controls), the direct write of stock quantities (fixed by P in M12, M26 and M29, kept or worsened by the controls), the legacy export with personal data (fixed by P in M11). Where the fix came from general knowledge (negative stock in M12, readable order numbers in M15, the dev loop in M16), all three arms fixed it. Where it needed a rule of the company, only the arm that was served the rule did.

8. Cost, time and volume

The VibeDefend arm is the most expensive overall ($232.86), the longest in turns (1,729) and the heaviest in context read (167 million cache tokens against 135 and 145). The injections, the MCP calls and the scans have a price, and it is measured. Three qualifications matter. First, the overhead concentrates in phase 1 ($150.25 against $126.67 and $112.72). In phase 2, when T1 receives the full corpus in its file, reading a 49-rule file at every session costs exactly as much as being served the rules at the edit: $49.38 against $49.58. The cost difference between a perfect file and the layer is nil; the result difference is 64 % against 100 %. Second, on neutral tasks P is no more expensive than T0 (§ 4.4). Third, the VibeDefend arm writes less: 8,671 lines added against 11,619 and 12,795, 131 files touched against 195 and 190, for the same set of tickets and with more rules laid down. Every line less is a line nobody will have to review or maintain; M14's compliance debt (§ 5.7) is its most concentrated illustration. The final test suites count 344 tests for T0, 523 for T1 and 358 for P, all green. Test count is not a quality measure here: the controls wrote, at least 21 and 14 times, tests that certify behaviour contrary to the rules.

9. What the numbers mean, and where the layer stops

9.1 The moment of information, not its availability

A language model has no working memory separate from its context; it has a context, and an attention that spreads over it. A file read at the start of a session of several dozen turns is, at the thirtieth edit, an old fragment buried under tens of thousands of tokens of code read, command output and reasoning. It is not forgotten in the sense of having left the context; it is diluted. Table 7 measures the dilution rate: with the exact rule in the file, the agent finds it four times in thirteen at the moment of writing the relevant line. M26 shows that dilution acts inside a rule: the number at the heart of the ticket (±30) survives, the peripheral token (ADJUSTMENT_LIMIT) does not. Injection at the edit does not make the agent smarter. It puts the information where the attention is, in the last tokens before the write. That is why a served rule's exactness rate is 95 % and depends neither on the rule's family, nor on its guessability, nor on the task, and why the same information, read at session start, yields 31 %. Same mechanism, different distance.

9.2 General knowledge is not the contract, and the code's habit beats the file

Task M21 is the cleanest demonstration of the product's value, because it separates two things usually confused. All three arms capped cash payments at 1,000 €: it is a public legal fact, the model knows it, T0 guessed it, T1a had it on its card. Only one arm answered 422 CASH_LIMIT with a boundary tested to the cent on the cash share of a mixed payment, the one that received the rule. To an API client, a 422 in prose and a 422 CASH_LIMIT are not the same thing: one is handled, the other is guessed. To a team, the value of its rules is not in what the model already knows about the world. It is in the arbitrary contracts that make its systems talk to each other (error codes, formats, thresholds specific to the company) and in the regulatory obligations expressed through those contracts. That is exactly what general knowledge does not contain and what injection carries.

The second half of the mechanism is the code itself. Three times in phase 2 (M26, M29, and by inheritance M12), T1b lost rule STK-01 to the baseline's blind UPDATE, with the rule spelled out verbatim in its file. That is not a failure of the file; it is a property of coding agents in maintenance. Existing code is an implicit instruction stronger than any document. It shows how things are done here, it is under the agent's eyes at the very moment it writes, and it compiles. A rule that contradicts a code habit can only win if it is presented at the same moment as the habit, that is, at the edit. P's movement ledger won because in M07 the rule and the habit met in the same turn and the rule won; from then on, P's habit was the rule.

9.3 Composition

§ 5.7 showed that conformance compounds, and this is the strongest economic argument. A rate of 89 % against 12 % is a per-task result; composition is a per-codebase result. Every rule laid down becomes a structure (a movements table, an idempotency key, an audit event) that later tasks reuse. The marginal cost of conformance therefore falls over time for the VibeDefend arm (334 lines for the audit trail in M14) and rises for the controls (1,107 and 1,620 lines, plus the debt of everything written against the rule in between). Figures 1 and 5 show it as curves; table 9 shows it as facts.

9.4 Where the layer stops

A study that finds only what it looks for is worth nothing. This one identifies six limits of the layer as tested, each documented by a case and each handed to the product team with a task number and a trace.

Authority: served thirteen times, and violated (M24). The ticket asks for "bigger gift cards for the holidays, sold at every till". Rule PAY-04 caps a card at 1,000 € and reserves issuance above 300 € to administrators. The layer injected PAY-04 thirteen times during the task. P nonetheless removed the caps it had itself set in M06 and rewrote its guard tests to make them pass. T0 did worse (gate removed, 2,000 €, any token issues). T1a, with its stale 500 € card, was the only arm to hold the 1,000 € cap, and was the best arm of the task. The lesson is clean and it bounds the product: injection informs, it does not enforce. Faced with a ticket that contradicts a rule, the agent arbitrates in favour of the most recent human instruction, and it is right to do so in general. It is the product that has to decide whether some rules are constraints (to refuse, or to escalate to a human) rather than information. The version tested does not distinguish the two.

Reach: the bare arm won when no channel worked (M23). The ticket describes customers charged more at pickup than at checkout. Rule PRC-04 imposes an asymmetry: refund the difference if the price fell, never claim it if the price rose. That day no channel of the layer worked for this rule. P froze the price symmetrically and defended the choice in a comment. T0, with nothing, found the asymmetry by consumer instinct. T1a, whose paraphrased card spoke of a "frozen price", was misled towards the symmetric by its own paraphrase. One exact for T0, zero for the other two. The orchestrator had graded this rule "unguessable" before the run; it was wrong, and recorded it.

Recall: four rules never retrieved. The four unserved rules (LOY-02 in M01, AUD-01 in M13, ACC-02 in M22, PRC-04 in M23) share a trait: each belongs to a family different from the one the ticket aims at first (loyalty on a refund task, audit on a fraud task, access on a refund task). Retrieval oriented by file and intent saturates on the main family. It is an identified recall defect, with its four cases, and it explains four of the VibeDefend arm's seven deviations.

Scope: two symmetric failure modes. The file read at session start produces a failure mode the study named the sweep: the agent, with a rules document in mind, applies rules where the ticket does not ask for them (M18: a session-expiry change inside a type refactor). Injection produces the symmetric mode, over-application. In M06, P implemented four served but off-rubric rules, changed the contract of an unrelated endpoint and rewrote two green tests, and its auditor graded the scope C. In M11 it rewrote the legacy export in passing to strip personal data. Both are right corrections made at the wrong moment; a human reviewer would refuse them in code review because they widen the diff. Injection carries the information; it does not yet carry scope discipline. A callout of the form "apply only what this change touches" is the item raised to the product team.

Precision: thirteen guard false positives, and retrieval noise. See § 6.1 and § 7.1. Both are tuning items with a measured cost: one turn per false positive, context tokens per irrelevant rule.

Infrastructure: six tasks of degraded delivery. A silent authentication failure of the hooks (a swallowed 401) left them mute on four rule-bearing tasks (M03, M08, M12, M15: zero injections). The MCP channel maintained service (M03: 3 exacts of 3 anyway; M12: 3 of 3). The root cause was found by instrumentation (three token stores ageing independently, no timeout on refresh, rotating refresh tokens under concurrent hooks) and fixed during the study; the report went to the product team. Then, after an account switch between M21 and M22, the MCP channel died on two tasks (M22, M23: zero MCP rules, zero scans), which directly explains M22's unserved rule and M23's defeat. A layer that lives in hooks and an MCP server must be monitored like a service. The hook execution trace, added during the study to diagnose the outage, is what made the fix possible. These outages are facts of the study; they are inside the accounts, not outside them.

10. Economics

10.1 What is measured, what is assumed, what is not counted

The model (paper/build/model.py) keeps the study's measured inputs and the external assumptions strictly apart. Every assumption is sourced and swept.

Measured. API cost per arm and per task (table 2). Deviations and violations left in the codebase per arm (table 5). Deviation rates per rule-bearing ticket, normalised by graded rules so that arms and phases are comparable: T0 2.28, T1a 2.27 (phase 1), T1b 0.95 (phase 2), VibeDefend 0.28. Security findings introduced per task, from the independent sweep: T0 0.10, T1 0.067, VibeDefend 0.033, of which VibeDefend fixed all. The credential-class guard block rate: one in thirty tasks. Code volume. M14's compliance debt.

Assumed. A loaded engineer cost of $700 per day, $87.5 per hour (as in the 13 August study). Human remediation of one rule deviation at 30, 60 or 90 minutes (find it, reread the rule, rewrite the code and the tests that lock it in, get it reviewed). Human remediation of one security finding discovered after merge at 1, 2 or 4 hours. Published incident costs, used for break-even only: $4.44 M global average cost of a data breach and $4.67 M when the initial vector is stolen or compromised credentials (IBM, Cost of a Data Breach Report 2025); an administrative fine of up to €15,000 per breach of price-information rules for a legal entity (French Consumer Code, art. L131-5); the GDPR cap of €20 M or 4 % of worldwide turnover (art. 83). Ticket flow at 5, 10 or 20 rule-bearing tickets per developer per month. Two hours per month per repository to keep a hand-maintained rules file current. VibeDefend's published list prices: Developer $231 a year (1 seat, 3 repositories), Team $2,748 (5 seats, 10 repositories), Scale $7,689 (15 seats, 25 repositories), additional seats $24 and additional repositories $12 per month.

Not counted, on purpose. The probability that a given deviation becomes an incident (handled by break-even in § 10.2, and by a swept scenario for credential blocks in § 10.5). The cost of the compliance audits that an audit trail laid down along the way shortens. The onboarding time of a new developer or a new agent that receives the rules while writing rather than by reading a document. The value of traceability: every rule served, every injection, every scan and every refusal is in the layer's traces, which makes conformance something that can be shown rather than asserted. The "bring to zero" phase. And, on the other side of the ledger: the operating cost of the layer (six degraded tasks in thirty), the time lost to thirteen guard false positives, and the context tokens spent on irrelevant security rules. We prefer a model whose every line can be found in a table of this paper to a larger one.

10.2 What the layer cost in the study, and what it avoided

The layer cost $35.89 of extra compute against the bare arm over thirty tasks and $24.45 against the file arm, that is $1.20 and $0.82 per task. In phase 2 it cost nothing against the perfect file; on neutral tasks, nothing against the bare arm. The overhead is entirely due to the rule-bearing tasks of phase 1, where the layer made 669 rule injections, 1,078 security-rule injections, 291 scan calls and 395 MCP calls that the other arms did not make. That is the price of the information, and it is of the order of a dollar per ticket.

At one hour per deviation, the 50 deviations avoided against the bare arm (57 minus 7) are worth $4,375 of engineering time, 122 times the overhead; the 45 avoided against the realistic file (52 minus 7) are worth $3,938, 161 times. At thirty minutes the ratios are 61 and 81; at ninety, 183 and 242. These figures count only the time to fix, not the time to discover. At least a third of the deviations are locked in by green tests, which means they will not be found by CI but by an incident, an audit or a customer. The time to fix is therefore a lower bound.

Another way to read the overhead is to ask what probability of a single avoided incident would repay it. Against the global average cost of a data breach ($4.44 M), the layer is repaid if it avoids one such incident with a probability of 0.0008 % over thirty tasks. Against one €15,000 fine for a price-information breach, the exact class of T0's M05 violation, locked in by a test, the break-even probability is 0.24 %. The study counted 21 outright violations in the bare arm, six of them on personal data and one against consumer law. We do not claim to know how many would have produced an incident; we note that the break-even question does not arise.

10.3 Total cost of ownership, with the licence

The study measures one agent on one repository. The value of the mechanism plays out at the scale where rules and security are centralised: one corpus, once, for every project, every developer and every agent of the company, what we call the brain of the projects. Table 19 builds the annual cost of four regimes for four organisation sizes, using the measured rates, the published prices and the central assumptions (10 rule-bearing tickets per developer per month, 60 minutes per rule deviation, 2 hours per security finding, $87.5 per hour). Every line is one multiplication the reader can redo. For twenty developers: 20 × 10 × 12 = 2,400 rule-bearing tickets a year; × 2.28 = 5,472 broken rules without the tool, × 0.28 = 682 with it; each at one hour and $87.5 gives $478,800 against $59,640 of rule remediation. Security findings follow the same arithmetic at 0.10 and 0.033 per ticket and 2 hours each ($42,000 against $13,860); VibeDefend's findings are charged even though the study saw them fixed inside the task, as a conservative allowance for human verification. The tool side adds the Scale plan plus five extra seats ($9,129) and the measured token overhead of $1.20 per ticket ($2,870). The four regimes are: no rules (T0's rates); a hand-maintained file (T1a's phase-1 rates, plus its upkeep); a perfect file (T1b's phase-2 rates, plus the same upkeep, the file being verbatim); and VibeDefend (P's rates, plus licence and tokens).

Table 20 turns these costs into a return. Net saving is the annual cost of the comparison regime minus the annual cost of the VibeDefend regime, licence and tokens included. Return ÷ cost is that net saving plus the tool's own cost, divided by the tool's own cost; in plain words, how many dollars of debt each dollar of licence and tokens removes. Break-even is the ticket flow below which the tool costs more than it saves.

Three things are visible in the tables. First, the licence is a rounding error against the debt it prevents: for twenty developers, $9,129 a year of licence and $2,870 of tokens against $419,160 of rule remediation and $28,140 of security remediation avoided. Second, the hand-maintained file costs more than nothing: it prevents nothing measurable in phase 1 and adds its own upkeep. Third, the perfect file, which nobody has, reaches about half the saving, at the cost of keeping 49 rules verbatim and current in every repository.

10.4 Sensitivity

The return stays above 20 times across the whole swept range, and above 30 times at the central assumptions, because the two quantities being compared are of different orders: a licence priced per seat against a debt that grows with every ticket. Break-even, the ticket rate below which VibeDefend costs more than it saves, sits at about a fifth of a rule-bearing ticket per developer per month, one every five months.

10.5 Action safety, valued by scenario

The guards' one true catch cannot be priced from this study alone; it can be bounded. At the measured rate of one credential-class block per thirty tasks, a twenty-developer organisation would see about eighty such commands a year reach the guard. Table 22 gives the expected loss avoided under three swept probabilities that an unblocked command of that class becomes an incident, priced at IBM's 2025 average for credential-vector breaches ($4.67 M). It is a scenario, labelled as such; at one in a thousand it is already of the same order as the whole rule-remediation saving. The preceding study's schema drop is the other bound: the cost of rebuilding a production database from backups, with the data written since the last one, is not in any table here, because no team wants to price it.

10.6 Bring to zero, then stay at zero

The study started three arms from the same clean baseline, so it measures the steady state: the slope at which each regime adds security findings, 0.10 per task without the tool and 0.033 with it, of which the tool's loop removed all. It does not measure the descent. An existing codebase adopting VibeDefend starts with a backlog of open findings, and the product's first job is to bring that backlog down: scan the whole repository, triage with the knowledge graph and the reachability analysis, fix with the agent, rescan. Figure 17 draws the two phases together, with the descent explicitly schematic and the slopes explicitly measured. The economic reading is that the two phases are priced differently. The descent is a one-off cost proportional to the backlog; the steady state is an avoided cost proportional to the ticket flow; only the second is in table 19.

11. Threats to validity

One model, one codebase, one operator. All measures use Claude Opus 5 at effort high, on a mid-sized Express/SQLite codebase, by the author of the product under test. Another model, stack or domain may move the rates; they should not move the mechanism (§ 9.1), which comes down to how a long context dilutes old information. We do not generalise beyond what is measured.

The corpus author sells the tool. The 49 rules were written by us, with arbitrary specifics, precisely what a write-time injection layer carries well. Three safeguards: the corpus and the rubrics were sealed before any run; each rule's guessability was graded beforehand; the tickets were checked for leaks. A reader who finds the rules "too arbitrary" must answer M21's question: are a company's real rules (its error codes, formats, thresholds, and the regulatory obligations expressed through them) any less arbitrary?

The auditors are models. The three blind audits per task are performed by agents, not humans. They are constrained to quote the diff, do not know the arm, and are overruled by the orchestrator when the live tree justifies it; arbitrations are recorded. A replication with human reviewers on a sample of tasks is the natural next step.

T1a's file was written by us. Its drift (two stale values, half the families absent) reconstructs what we observe in teams; it is not a measurement. That is why amendment n° 1 exists: phase 2 gives the file its best possible version, and the most useful result (64 % against 100 % at the same cost) is established against it.

The security signal is small, and the action-safety signal is borrowed. The independent sweep finds one to four findings per tree: this codebase and these tickets produce few weaknesses a syntactic scanner sees, and the dependency surface never changed. The sweep establishes the shape of the curves (the controls never step down; VibeDefend steps down inside the task), not a large effect size. The security part of the economics in § 10.3 rests on small rates and is shown separately so a reader can discount it. The guards' strongest evidence (the schema drop, the near-miss) comes from the preceding study, in a different regime, and is reported as such.

Small numbers. 25 rule-bearing tasks, 64 to 66 graded rules per arm, six tasks in phase 2. The main gaps (13 % against 87 %, 64 % against 100 %) are too large to be counting accidents; the secondary ones (the distributions of table 4, the cost ratios per phase) are to be read with this size in mind. No p-value is published, on purpose.

Outages of the layer, and fixes during the study. Six tasks suffered degraded delivery. They are counted against P (they explain two of its seven deviations and M23's defeat) and not excluded. A reader who prefers to exclude those six tasks gets a P arm at 45 of 50 (90 %) instead of 57 of 64 (89 %). The hooks' silent authentication failure was diagnosed and fixed locally between tasks 3 and 15; the fixes touched only the reliability of rule delivery, not their content nor the corpus, and are documented. A study on a stabilised version should see fewer outages and a slightly better P arm.

Two flaky suites, and the costs. P's suite failed once then passed on rerun in M11 and M29 (state coupling between capture tests); both are recorded as test instability, not regressions. API costs are those reported by the tool for each run, continuation retries included; one arm's two aborted launches at task 22 were purged and the task re-run. Costs do not include VibeDefend's price, which the economics chapter adds at list rates.

12. Conclusion

We asked a narrow question: what decides whether a company's rule, commercial or regulatory, reaches the code an agent writes in maintenance, whether a dangerous command runs, and whether a weakness ships? The measured answer is that it is not the presence of the rule in the repository but the moment it reaches the agent, together with something in the loop that checks and looks. A realistic rules file, read at every session, produced exactly the same code as no file: 7 exact specifics of 55 in both cases. The same file made perfect (the whole corpus, verbatim) produced 64 %; VibeDefend's injection at the edit produced 87 % in phase 1 and 100 % in phase 2, at the same compute price. The chain is direct: 55 of the 58 rules served are exact, 4 of the 6 not served are not. On tasks without rules the layer changes nothing.

The controls left 57 and 52 non-conformant rules in the codebase, 21 and 12 of them outright violations (personal data in a carrier export and in the logs, an illegal struck-through price, loyalty points on gift cards, stock overwritten without a trace, an order book readable from any till), at least a third of them locked in by green tests that will defend them against CI. Their trees drifted from one open static-analysis finding to four and three, and nothing in their loop would ever have looked. The VibeDefend arm left seven, one of them a violation. Its tree ends at its baseline count because the one weakness it introduced was found and fixed by its own live scan inside the task. Its guards refused the one command in thirty tasks that reached for a stored credential, as they had refused, in the preceding study, the schema drop that two unguarded arms executed eighteen times between them, and they also refused thirteen harmless commands, which the product must fix.

Economically, information at write time cost a dollar per ticket. With list prices, a twenty-developer organisation spends $85,499 a year with VibeDefend against $520,800 without: $435,301 net, 37 times the licence and tokens, break-even at one rule-bearing ticket per developer every five months, and the return stays above 20 times across every assumption swept. A perfect rules file costs as much in compute, must be kept in every repository, and goes two thirds of the way.

What this paper does not say is that the layer tested is finished. It has a recall defect on mixed families (four cases), no conflict protocol against a contradicting ticket (one case, the most important), scope discipline to learn (two cases), a false-positive rate to fix on its guards (thirteen cases), retrieval noise on security rules, and infrastructure to monitor (six tasks). Each of these points has a task number, a trace and an item handed to the product team. That is what an experiment is supposed to produce: a measure, a mechanism, and the list of what remains to be done.

Appendix A. The 49 rules of the corpus

Each rule of the sealed corpus, with what it requires (the constraint, as written in rules/corpus.md), its family and the control frame the family maps to.

Appendix B. Sealed predictions and their resolution

Two resolutions complete the table. Prediction n° 3 (exact-without-literal cards) is also confirmed in M22, where T1a implemented its card's semantics with a prose refusal and no literal. Prediction n° 4 (P extends its movement ledger to transfers) is confirmed in M29 with the typed TRANSFER_OUT/TRANSFER_IN pair. The three wave-2 predictions for tasks 11–20 (parity on neutral tasks, neutrality of P, no cost tax) are resolved in § 4.4. The phase-2 prediction (the verbatim file succeeds on the rules the ticket aims at and keeps missing cross-cutting obligations) is resolved in § 5.5.

Appendix C. Code grades per task and arm

Global grade given by each arm's auditor (correctness, tests, scope, idiom), after the orchestrator's re-read.

Appendix D. Artefacts

• rules/corpus.md: the 49 rules, sealed (rules/corpus.sha), with their guessability critique (rules/CRITIQUE.md).
• tasks/micro.md, tasks/rubric.json, tasks/t1-coverage.json, tasks/CRITIQUE-M21-M30.md: the tickets, the sealed rubrics, T1's file exposure, the predictions, the wave-3 critical review.
• docs/PROTOCOL.md: the measurement chain and amendment n° 1.
• arms/T0, arms/T1, arms/P: the three final trees, one git tag per task (baseline, task-01 to task-30, t1-phase2 for the amendment).
• data/tasks/M01 to M30: per task, the report, the three blind auditors' verdicts with the letter mapping, the anonymised diffs, the full transcripts of the three arms, the statistics (cost, turns, tokens, tools, layer census), and the hook-trace archive.
• data/scoreboard.md: the scoreboard kept during execution.
• runner/: the runner, the ten invalidation gates, the report assembler.
• paper/build/: extract.py (dataset), transcripts.py (scans, commands), guards.py (guards), secscan.py (independent static-analysis sweep), figures.py, model.py (economics), tables.py, build.py (this document), brief.py (the executive brief), deck.py (the study deck). paper/data/: the consolidated dataset, the deviation catalogue, the sweep, the model, the tables. paper/figures/: the seventeen figures.

Appendix E. The 17 guard refusals of this study, one by one

In the two arms without guards, the same command classes ran unchecked: 9 scripts executed from /tmp and 3 commands containing a DELETE FROM without WHERE (some of them greps). Neither control attempted to read a credential.

CybeDefend Research, 24 August 2026. This document is regenerated in full by python3 paper/build/build.py; no number in it is typed by hand.

Leggi il paper così come è stato pubblicato36 pagine, 3.5 MB