Think-briefly instruction vector, self-consistent compile (dose 2): each layer writes only what the paragraph's mean shift still lacks
When
Recorded date: 2026-11-14
Ledger date only; exact weight-build/change timestamp is not recorded here. Not a live training event.
Exact tensor-change timestamp: unavailable.
Why change it?
What the last round showed (batch x001, 120 stage-1 tasks, counted from `task_results.jsonl` by `code/x_stats.py` and `code/x_seq.py`). 0031 `icv-brief-b20` and 0032 `icv-brief-b40` write, through an always-on neuron in every layer, the mean stream shift of a paragraph "keep your thinking very short ..." at doses 2 and 4. | arm | solved | at the 10,000-token budget | empty patch | finished with a wrong patch | output tokens per episode, median | |---|---|---|---|---|---| | unedited (two arms) | 72, 64 | 45, 46 | 21, 18 | 16, 19 | 8,410, 7,846 | | 0031, dose 2 | 77 | 32 | 13 | 17 | 6,996 | | 0032, dose 4 | 71 | 19 | 8 | 31 | 5,308 | So the…
Which part changed?
Think-briefly instruction vector, self-consistent compile (dose 2): each layer writes only what the paragraph's mean shift still lacks
The source describes adapter targets and mathematical changes. No before/after tensor comparison is available in this viewer.
How was it changed?
Compile the “think briefly” instruction contrast into one neuron per layer, but account for changes already written by earlier edited layers: each layer writes only the remaining mismatch. This is the self-consistent version rather than independently adding the original difference at every layer.
Did it improve?
null · Stage 1: -2.00 (x004) tasks against reference arms. Stage 2: not yet measured. Matched control, stage 2: not yet measured. Verdict: null.
Evaluation set: wsp31_screen_v2 · Batch: x004. Local screening and historical task scores are not competition scores unless explicitly identified in the source. Different evaluation sets cannot be compared as a gain.
Original evidence
Literal source excerpts follow. They may contain original predictions and superseded counts; use the measured-outcome summary and corrections above for the current result.
Hypothesis
What the last round showed (batch x001, 120 stage-1 tasks, counted from `task_results.jsonl` by `code/x_stats.py` and `code/x_seq.py`). 0031 `icv-brief-b20` and 0032 `icv-brief-b40` write, through an always-on neuron in every layer, the mean stream shift of a paragraph "keep your thinking very short ..." at doses 2 and 4. | arm | solved | at the 10,000-token budget | empty patch | finished with a wrong patch | output tokens per episode, median | |---|---|---|---|---|---| | unedited (two arms) | 72, 64 | 45, 46 | 21, 18 | 16, 19 | 8,410, 7,846 | | 0031, dose 2 | 77 | 32 | 13 | 17 | 6,996 | | 0032, dose 4 | 71 | 19 | 8 | 31 | 5,308 | So the compiled paragraph does in whole episodes what it says, in dose order, and dose 2 solved more than either reference arm (+9.0 by the rule; +5 against one arm and +13 against the other, two identical arms differing by 8). At dose 4 the saved tokens became wrong patches, and five of the six tasks it lost ran to the budget in reads without an edit. Two further facts bound what this lever can give. (1) The tools lane's budget run (`results/wsp31/budget/s1_x124/summary.txt`, unedited model, the 67 stage-1 tasks whose unedited episode ended on a cap): 22 solved at 10,000 tokens, 27 at 20,000 (gained 8, lost 3; the 40,000-token arm failed on sandbox errors). Doubling the budget is worth about +5 of 120, so saved tokens cannot be worth more. (2) In all four arms of x001 the wrong patches fail the required tests and nothing else (failed tests = required tests in 125 of 136 non-empty unsolved episodes and in 79 of the 83 that finished on their own; exactly one failed test in 73), and the agent had run a test after its last edit in 71% to 88% of the finished-wrong episodes, about as often as in solved ones (77% to 86%). The remaining failures are fixes that do not do what the issue asks; they are not regressions and not a lack of testing. This entry asks one question: **is the gain of 0031 real?** It is the same instruction at the same dose, rebuilt with an independent neuron in every layer and a second way of compiling, scored in a new batch against new reference arms. It is also the within-batch reference for its sister entry `fcv-brief-exact-b20` (this edit with a second paragraph compiled in, on being exact about what the issue asks: the difference between the two entries in one batch is the effect of that paragraph). The other compile. Turn 6 wrote `delta_L = d_L - layer_scalar_L d_(L-1)` (d_L = the paragraph's mean shift of the stream after layer L), which assumes a layer passes a shifted input through unchanged. Here the write of layer L is solved for directly: what is still missing at layer L once all layers below already carry exactly the wanted shift (one hooked forward pass per call, recipe below). I expected more from this than it gave. My reading before the measurement was that half of what 0031 puts into the stream was off-target (cosine 0.4 to 0.5 with the prompted shift) and that a self-consistent compile would remove it. Measured (sampled layers above layer 0): the self-consistent write is 0.5 to 0.95 times the size of the turn-6 write at a cosine of 0.42 to 0.84 with it; on held-out calls the stream shift at dose 2 is 1.2 to 1.9 times the paragraph's (turn 6: 1.9 to 2.5; intended: 2.0), so the new compile undershoots where the old one overshot slightly; **and the cosine with the prompted shift stays 0.41 to 0.71**. The missing half is not a compile error: it is the part of the paragraph's effect that differs from call to call, which no constant vector carries. On the logged path the two compiles are the same edit to within the pre-test's resolution (KL from base 0.0075 against 0.0072; 12 against 10 of the 15 prompted top-token changes matched; table below). So this entry is a replication of 0031 at about 0.7 of its stream shift, not a different edit, and I predict accordingly. Different from the null and harmful rows it resembles: the six close-the-thought edits (0008, 0011, 0012, 0020, 0021 harmful; 0015 +4.5) push the end-of-thought token or state; this pushes nothing towards `<channel|>` (log p(close) at logged closes falls slightly, as under the paragraph itself). Against 0031 itself: the fourth-quietest neuron of each layer instead of the third, the solved write instead of the difference formula, a new batch. Risks: the gain of 0031 was noise (then this entry scores inside the reference spread); the ceiling of the lever is about +5 (budget run), under the +10 an edit needs. Champion. Built and tested on generation 0. If 0031 is promoted while this entry waits, the runner serves both always-on neurons together (different neurons, so the sum is well defined): the paragraph at about dose 4, which is 0032's regime (+3.0, wrong patches up). That stack is not intended; on a champion that holds 0031 this entry should be read as a dose test and is expected to do nothing or harm. Fitted on (task, unedited arm of batch b001, call index; all stage-1 tasks, none in a stage-2 list; the 35 calls of turn 6): mewwts__addict.75284f95.func_basic__9sg9rq7f base 3; scitokens__scitokens-193 aa 5; benoitc__gunicorn.bacbf8aa.func_basic__3napqcbj zero 3; rasbt__mlxtend-1087 zero 22; r1chardj0n3s__parse.30da9e4f.func_pm_remove_cond__k4df18dk zero 1; ross-rotordynamics__ross-177 aa 4; lepture__mistune.bf54ef67.combine_module__dalnbilu aa 3; openmdao__openmdao-3315 aa 22; prettytable__prettytable.ca90b055.func_basic__1mmuxqev zero 3; cool-RR__PySnooper.57472b46.lm_rewrite__031xvy9b zero 3; wireservice__csvkit-800 aa 11; life4__textdistance.c3aca916.combine_file__ciwv50c6 base 2; ishepard__pydriller-84 zero 2; zopefoundation__zope.configuration-34 base 26; stingraysoftware__stingray-234 base 29; jsvine__pdfplumber.02ff4313.func_pm_remove_assign__1v9soh1j zero 3; sqlfluff__sqlfluff-5362 zero 24; prometheus__client_python-1134 base 3; Textualize__textual-1352 zero 19; ipython__traitlets-517 aa 28; django-money__django-money.835c1ab8.combine_module__bqggrnha aa 13; pallets__jinja.ada0a9a6.func_basic__2vdao9be zero 27; geographika__mappyfile-146 zero 21; pypa__hatch-1148 aa 13; gcaufield__MonkeyPack-27 base 2; hdmf-dev__hdmf-1270 base 15; tophat__syrupy-768 zero 2; rubik__radon.54b88e58.func_basic__8pbqxg0r zero 8; juju-solutions__charms.reactive-71 zero 32; manicmaniac__sqlalchemy-repr-7 base 1; keis__base58-46 base 3; urbandataanalytics__SwissKnife-12 base 3; pytask-dev__pytask-200 zero 27; conda__conda-6131 aa 31; mewwts__addict.75284f95.combine_file__3fttj8ti zero 1.
Exact recipe
This entry: arm `F20` of `code/fcv_job.py` (paragraph I1, dose beta = 2). Control: arm `F20c`. base: champion generation 0 No edit had passed when this entry was built; statistics, weights and pre-tests are those of the unedited model. size: whole network 0.6753% over 180 matrices, largest matrix 1.6391% control size: whole network 0.6753% over 180 matrices, largest matrix 1.6391% The size is mostly the removal of the 60 replaced neurons (old gate and up rows of norm about 1); the functional dose is beta. Rank 1 per matrix, 180 matrices (gate, up and down of the MLP in all 60 layers). Built in the first GPU-0 hold of turn 8 (`flock gpu0.lock timeout 1500 code/gpu_hold8.sh`, 13 minutes; the turn's four holds took 13, 16, 6 and 2 minutes; forward pass `tools/g31.py`, no server, no optimiser anywhere) and rebuilt bit for bit on CPU by `build.sh` (`code/fcv_build.py` from `code/data/fcv_stats.pt`; the sha256 of `edit/` and `ctrl/` equal those of the arms the job evaluated). Paragraph I1 (56 tokens with the blank line before it; inserted at the end of the first user message): > Important: keep your thinking very short. Before each tool call, think in at most three short sentences: what you just learned and what you will do next. Never write long plans, never draft or rewrite code in your thoughts, and never go over the same point twice. Data: as in 0031. `tools/data/probe_calls_b001_full.jsonl`, stage-1 tasks in no stage-2 list, calls after the first of an episode, at most 4,600 tokens, a closed thought and 24 or more completion tokens, one call per task; every fourth task (sorted by name) held out; the fit calls are the 35 of turn 6, in the same order. No task outcome enters any statistic. Target. `d_L` = mean over the fit calls of the mean over completion positions P (at most 300 per call, evenly spaced) of `resid_post_L(with the paragraph) - resid_post_L(without)`: taken unchanged from turn 6 (`code/data/icv_stats.pt`, `code/icv_job.py`). Self-consistent write (`fcv_job.capture`). For each fit call: one unedited pass that stores `resid_post_L` at P for all 60 layers, then one pass with a hook after every layer L, in layer order: `s_L = mean over P of (resid_post_L of this pass - stored resid_post_L)`; `c_L = beta d_L - s_L`; `c_L` is added to `resid_post_L` at every position except BOS. After the hook the mean shift over P is `beta d_L` exactly, so the next layer sees the shifted input it would see under the instruction, and `c_L` is only what layer L and the residual path did not supply. `F_L = mean over the 35 calls of c_L` (dose 2 has its own pass; the response is not linear in beta). Layers [0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 59]; `delta6` = the turn-6 write `d_L - layer_scalar_L d_(L-1)`: | dose | size of the write over dose x size of delta6 | cosine with delta6 | split-half cosine (even against odd fit calls) | |---|---|---|---| | 1 | 1.00 0.72 0.58 0.95 0.75 0.73 0.96 0.67 0.76 0.84 0.92 | 1.00 0.54 0.43 0.72 0.66 0.64 0.82 0.69 0.68 0.66 0.48 | 1.00 0.99 0.93 0.88 0.95 0.87 0.96 0.86 0.82 0.80 0.90 | | 2 | 1.00 0.67 0.51 0.92 0.72 0.70 0.95 0.65 0.72 0.80 0.87 | 1.00 0.56 0.42 0.76 0.61 0.66 0.84 0.73 0.72 0.67 0.48 | 1.00 0.99 0.90 0.90 0.90 0.92 0.98 0.92 0.91 0.90 0.95 | | 3 | 1.00 0.67 0.45 0.83 0.72 0.71 0.94 0.64 0.71 0.79 0.88 | 1.00 0.57 0.43 0.79 0.62 0.65 0.84 0.73 0.72 0.68 0.49 | 1.00 1.00 0.97 0.92 0.96 0.96 0.99 0.95 0.94 0.93 0.96 | Always-on neuron (one per layer, replaced outright; the code of `icv_job.fit` with one change). The neuron: **fourth** smallest mean |activation| in layer L in my turn-3 capture (`scratch/job3/net_stats.pt`; 0031 used the third, so the two edits touch different neurons). Read vector `w_L`: ridge fit of the constant 1 on the MLP input at completion positions of the fit calls with index % 3 != 0 (dual form, lambda = mean squared coordinate; chosen over the mean input in 60 of 60 layers by its smaller sd on the calibration calls), scaled to mean 1; no BOS projection was needed (0 of 60 layers). Gate row = 8 `w`, up row = `w`; `gbar_L` = mean activation over completion positions (bf16-rounded rows), 7.8 to 8.0. Coefficient of variation on the calibration calls 0.05 to 0.12 at the 11 layers printed in the job log (`scratch/logs/job8.log`). Write. Down column `c = clipdiv(F_L, w_post_L) * rmsy_L / (gbar_L * layer_scalar_L)`: `w_post` = weight of the post-feedforward norm, `rmsy_L` = mean RMS of the raw MLP output at completion positions, `clipdiv` divides coordinate-wise and zeroes the 5% of coordinates with the smallest |w_post| (`code/thinklib.py`). The write relative to the RMS of the raw MLP output (`rho` in `edit/edit.json`): 0.004 to 0.051, median 0.014. Intended against built. Intended: after every layer the mean stream shift over completion positions equals twice the paragraph's. Built: a constant per layer (times the neuron's activation, about constant) solved for that on 35 calls with the correction applied as an exact vector by a hook; the served neuron writes through the post-feedforward norm, fires at 0.75 to 0.93 of `gbar` on prompt positions and at 0.25 to 3.3 on BOS, and writes one vector for all calls. Measured result on held-out calls: stream shift 1.24 to 1.93 times the paragraph's (layer 0: 3.99) instead of 2.0, cosine 0.41 to 0.71 (list below). Not built: anything of the paragraph's effect that depends on the call or the position. Control: same neurons, same gate and up rows, and the written vector `F_L` with the sign of every coordinate flipped at random (seed 6000 + L) before the same `clipdiv` and scaling: identical per-coordinate magnitudes, so the same |dW|_F and the same size of write, and no relation to the instruction. It is the right control because it holds the removal of 60 neurons and an always-on write of the same size in every layer, and differs only in direction. ### Pre-test (forward pass, teacher-forced on the logged path; 8 held-out calls) `icv_job.evaluate` (my script of turn 6, not checked against another tool). Each held-out call is run unedited (base), unedited with the paragraph (prompted) and with each adapter; numbers are at the positions of the logged completion, against base. "flips" = positions with another top token; "close on top inside" = thought positions where `<channel|>` becomes the top token; "dlogp close" = mean change of log p(`<channel|>`) where the agent went on / where it closed; "reproduced" = 1 - KL(prompted || arm) / KL(prompted || base); "shift cos" = cosine between the arm's and the prompt's change of the log-probs of base's 50 top tokens; "prompt flips matched" = of the positions where the prompted top token differs from base, those where the arm's top token is the prompted one. Eight calls: read 0.05 of "reproduced" or 2 flips as nothing. | arm | calls / positions | flips | flips in thought | close on top inside | dlogp close inside | dlogp close at close | tool flips | KL from base | reproduced | shift cos | prompt flips matched | |---|---|---|---|---|---|---|---|---|---|---|---| | prompted with the paragraph (the target) | 8 / 1278 | 0.012 | 0.020 | 0 of 460 | -0.07 | -0.192 (7 of 8 kept) | 0 of 8 | 0.0076 | - | - | - | | self-consistent compile, dose 1 (F10) | 8 / 1278 | 0.009 | 0.013 | 0 of 460 | -0.07 | -0.037 (7 of 8 kept) | 0 of 8 | 0.0022 | 0.47 | 0.34 | 7 of 15 | | **this entry**: self-consistent compile, dose 2 (F20) | 8 / 1278 | 0.014 | 0.024 | 0 of 460 | -0.24 | -0.101 (7 of 8 kept) | 0 of 8 | 0.0075 | 0.48 | 0.38 | 12 of 15 | | **its control**: control of F20 | 8 / 1278 | 0.009 | 0.017 | 0 of 460 | -0.12 | -0.001 (7 of 8 kept) | 0 of 8 | 0.0011 | -0.10 | 0.11 | 6 of 15 | | self-consistent compile, dose 3 (F30) | 8 / 1278 | 0.018 | 0.030 | 0 of 460 | -0.28 | -0.209 (7 of 8 kept) | 0 of 8 | 0.0172 | -0.09 | 0.38 | 12 of 15 | | turn-6 compile, dose 2 (0031 `icv-brief-b20`) | 8 / 1278 | 0.013 | 0.022 | 0 of 460 | +0.05 | -0.171 (7 of 8 kept) | 0 of 8 | 0.0072 | 0.45 | 0.34 | 10 of 15 | Stream shift (mean over completion positions of `resid_post`, arm minus base, against prompted minus base; layers [0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 59]): - self-consistent compile, dose 1 (F10): cosine 0.25 0.70 0.52 0.41 0.43 0.47 0.53 0.48 0.47 0.40 0.33; norm ratio 1.97 0.92 1.09 1.05 0.78 0.81 0.82 0.77 0.78 0.76 0.75 - self-consistent compile, dose 2 (F20): cosine 0.25 0.71 0.55 0.43 0.45 0.48 0.57 0.52 0.50 0.45 0.41; norm ratio 3.99 1.76 1.83 1.93 1.42 1.52 1.56 1.39 1.35 1.24 1.28 - control of F20: cosine -0.04 -0.00 0.05 -0.03 0.02 0.06 0.04 0.04 0.10 0.10 0.09; norm ratio 3.95 1.58 1.46 1.02 1.02 1.22 1.25 1.04 1.07 1.02 1.33 - self-consistent compile, dose 3 (F30): cosine 0.25 0.72 0.55 0.45 0.45 0.50 0.58 0.51 0.51 0.45 0.43; norm ratio 5.97 2.60 2.62 2.48 2.21 2.28 2.34 2.05 1.96 1.79 1.82 - turn-6 compile, dose 2 (0031 `icv-brief-b20`): cosine 0.25 0.68 0.51 0.46 0.46 0.48 0.54 0.45 0.46 0.42 0.37; norm ratio 3.98 2.04 2.37 2.18 1.87 2.18 1.88 2.27 2.12 1.94 2.48 Activation of the always-on neuron over `gbar` in the edited pass (dose 1, held-out calls, the same 11 layers): completion positions mean 0.94 to 0.99, sd 0.06 to 0.12, min 0.53, max 1.58; prompt positions mean 0.75 to 0.92; BOS 0.25 to 3.29. ### Served generation (vLLM through `tools/gpu0_session.sh`, greedy, single turns, no rollout) `code/icv_gen.py`: mid-episode calls of the held-out tasks (one per task, prompts of at most 4,600 tokens, so calls with short thoughts), answers of up to 1,500 tokens. "shorter / longer" = contexts whose thought is shorter / longer than the unedited model's; "cut" = answer not finished in 1,500 tokens; "loops" = answers of 600 tokens or more with more than half repeated 8-token sequences. The sister entry's arms are in the same table. | arm | thought tokens mean | median | shorter / longer | answer tokens mean | same tool as unedited | well-formed tool call | cut | loops | |---|---|---|---|---|---|---|---|---| | unedited | 69 | 30 | - | 135 | - | 16 of 17 | 0 | 0 | | unedited, paragraph I1 in the prompt | 35 | 25 | 8 / 0 | 116 | 15 of 17 | 16 of 17 | 0 | 0 | | **this entry**: self-consistent compile, dose 2 (F20) | 48 | 30 | 4 / 4 | 128 | 15 of 17 | 17 of 17 | 0 | 0 | | paragraph pair, dose 2 (G20) | 42 | 30 | 6 / 3 | 121 | 15 of 17 | 16 of 17 | 0 | 0 | | paragraph pair, dose 1 (G10) | 47 | 30 | 3 / 2 | 137 | 14 of 17 | 17 of 17 | 0 | 0 | | self-consistent compile, dose 3 (F30) | 36 | 22 | 9 / 2 | 101 | 15 of 17 | 17 of 17 | 0 | 0 | 17 contexts (calls 2, 17, 4, 17, 9, 3, 12, 12, 21, 26, 2, 13, 12, 17, 4, 26, 15). `interp/toolcall_check.py` on that file: 0 unpaired string delimiters in the 17 tool calls of every arm (UNDECIDED: it needs 60 calls); no answer reaches 1,500 tokens, so no loop rate. `tools/closeprobe.py` (40 logged closes and 126 goes-on points of the default per-call set; fourth hold): change of log p(close) at close +0.033 (raised at 33 of 40), at goes-on points -0.069; `<channel|>` is the top token at 40 of 40 closes. The tools lane's rule (below -0.05 at close: expect longer ordinary thoughts) does not flag it. On the 8 closes of the table above the same quantity is -0.10 (0031: -0.17 there, and its rollout thoughts got shorter): the two samples disagree in sign, so neither says more than "no large change". Not measured: whole episodes; thoughts beyond 1,500 tokens; tool-call syntax on 60 or more calls; generation speed under this adapter (rank 1 on 180 matrices, the same shape as 0031, which ran at 24.6 against 25.1 tokens per second). Checker: `check_adapter.py` passes on `edit/` and `ctrl/` (run by `build.sh`): 180 modules, r = 1. Runner facts (2026-11-14 01:39:23 EST): source `wsp31/authors/claude/queue/fcv-brief-b20/`; served copy `wsp31/queue/frozen/claude__fcv-brief-b20/`; sha256 over files, edit `e5be4842c9d805e0aa99ed185fbd2c826e54b33619c55a1e34a07d99c8f5acb5`, control `869323ea355d90944426b3721e683d94b16bd27867e58fd6099abfcaeccfb55a`; both passed the checker.
Prediction (written before the run)
predicted_gain_120: -2 to 8 My median is 3. 0031 measured +9.0 against reference arms that differ by 8 from each other; the budget run puts the ceiling of saved tokens near +5; this entry is 0031 at about 0.7 of its stream shift. I expect the measured gain of 0031 to shrink on replication. Behaviour that should move (edit against the mean of the two reference arms of the same batch, from `task_results.jsonl` with `code/x_stats.py`; x001 values of the references and of 0031 in brackets): - episodes that end on the token budget: 6 or more fewer (45.5 -> 32); - median output tokens per episode: lower than both reference arms, by 5% or more against their mean (8,128 -> 6,996); - empty patches: 3 or more fewer (19.5 -> 13); - finished episodes with a wrong patch: within 5 of the reference mean (17.5 -> 17); - median tool calls unchanged within 2 (18 -> 18). Control (sign-scrambled write of the same size on the same neurons): none of these; budget endings within 5 of the reference mean, gain -4 to 3. The hypothesis is wrong if (a) budget endings do not fall by at least 4 and output tokens per episode are not lower than in both reference arms: then what x001 showed for 0031 was not the instruction (noise, or something particular to its neurons or its compile); or (b) they fall as predicted and the gain is 0 or less: then the saved tokens do not turn into solved tasks, 0031's +9 was the reference arms' spread, and brevity is closed as a lever for me. A gain of +3 or more together with (a) would not be this mechanism. If this entry and 0031's stage 2 disagree, stage 2 (218 tasks, with a control) is the better measurement.
Result
**Stage 1** (batch `x004`, 4 arms on one server, concurrency 4 per arm, finished 2026-11-14 07:11:42 EST; `results/wsp31/batches/x004/`). Reference arm: unedited. Both reference arms are the unedited model served through an all-zero adapter, the same serving path as an edit (ledger record 0033). Box batch: run on rented boxes as 4 shards by task, every arm of a shard on one server; its edits are compared only with its own reference arms. Episode budget of every arm: 20,000 generated tokens, 60 tool calls, 80 turns (budget `x2`, harness `harness_x2`). - unedited: solved 78 of 120, agent errors 18, at session limit 18, at tool-call cap 4, empty patch 7, median tool calls 22.0, mean 388.4 s, median 24.3 completion tokens/s of model time. - edit: solved 75 of 120, agent errors 11, at session limit 11, at tool-call cap 2, empty patch 5, median tool calls 19.0, mean 342.2 s, median 23.9 completion tokens/s of model time; vs base on 120 common tasks: -3.0 tasks (95% interval -11.0 to +5.0), base-only 9, arm-only 6, exact p 0.6072. - second unedited arm (identical weights): solved 76 against 78; 7 tasks solved only by the first, 5 only by the second (12 tasks differ; record 0003 measured 11.3 per identical pair); median 24.6 completion tokens/s. The gain used by the rule is against the mean of the two reference arms per task. - edit against the mean of the two reference arms: -2.0 tasks of 120 (95% interval -8.5 to +4.5). - Predicted gain -2 to +8; measured -2.0: inside the interval.
Verdict and notes
null by the fixed rule: stage-1 gain -2.0, bar +3, harmful at -6 or below.