GEMMA DEVELOPER AGENTSAVED EVIDENCE

← Weight change

Direct mathematical edit / wsp31-0031

Instruction vector, think briefly: an always-on neuron per layer writes the stream shift of a 'keep your thinking short' paragraph (dose 2)

Gemma 4 31B, frozen four-bit baseAuthor: claudeState: doneVerdict: null

When

Recorded date: 2026-11-13

Ledger date only; exact weight-build/change timestamp is not recorded here. Not a live training event.

Exact tensor-change timestamp: unavailable.

Why change it?

Unedited arms of batch b004 (`code/arm_stats.py`, 102 episodes each): the median thought is 66 and 70 tokens per call but the mean is 234 and 240, because a fifth of the calls (19.6% and 21.1%) carry a thought of 300 tokens or more; an episode writes a median of 6,810 and 7,182 completion tokens in a median of 20 calls, so thought is the larger part of what it writes; 24 and 23 episodes end at the session limit, which the interpretability lane showed to be a budget of generated tokens. Six screened edits (0008, 0011, 0012, 0015, 0020, 0021) pushed the end-of-thought token or state, at the end of a thought or deep inside one. None lowered the…

Which part changed?

Instruction vector, think briefly: an always-on neuron per layer writes the stream shift of a 'keep your thinking short' paragraph (dose 2)

The source describes adapter targets and mathematical changes. No before/after tensor comparison is available in this viewer.

How was it changed?

Run the same calls with and without a “think briefly” paragraph, measure each layer’s mean activation shift, then replace one quiet neuron in every layer with a nearly always-on writer of that shift. This changes gate/up rows and down columns in the feedforward blocks across all 60 layers; dose 2. The runtime prompt is unchanged.

Did it improve?

null · Stage 1: +9.00 (x001) tasks against reference arms. Stage 2: +9.50 (b008). Matched control, stage 2: +7.50. Verdict: null.

No reliable improvement: recorded verdict is null.
Correction records supersede original result counts. The current summary uses the corrected paper table; uncorrected text below is original historical evidence, not the latest result.

Evaluation set: wsp31_screen_v2 · Batch: x001, b008. Local screening and historical task scores are not competition scores unless explicitly identified in the source. Different evaluation sets cannot be compared as a gain.

Original evidence

Literal source excerpts follow. They may contain original predictions and superseded counts; use the measured-outcome summary and corrections above for the current result.

Hypothesis
Unedited arms of batch b004 (`code/arm_stats.py`, 102 episodes each): the median thought is 66 and 70 tokens per
call but the mean is 234 and 240, because a fifth of the calls (19.6% and 21.1%) carry a thought of 300 tokens or
more; an episode writes a median of 6,810 and 7,182 completion tokens in a median of 20 calls, so thought is the
larger part of what it writes; 24 and 23 episodes end at the session limit, which the interpretability lane showed to be a budget of generated tokens.

Six screened edits (0008, 0011, 0012, 0015, 0020, 0021) pushed the end-of-thought token or state, at the end of a
thought or deep inside one. None lowered the tokens per episode; mine (0008, 0020, 0021) turned the thoughts that
still got long into loops. This entry pushes nothing towards `<channel|>` and has no gate. It writes, at every
position and in every layer, the mean shift of the residual stream that an explicit instruction paragraph causes in
the unedited model itself:

> Important: keep your thinking very short. Before each tool call, think in at most three short sentences: what you just learned and what you will do next. Never write long plans, never draft or rewrite code in your thoughts, and never go over the same point twice.

An instruction acts through attention to its tokens, which moves the stream at every later position; the part of
that movement that is the same at every position is a constant vector per layer, and a constant vector is what an
always-on neuron can write (the model has no biases). If the shift carries the instruction, the model plans a short
thought from its first token, instead of having a long one cut off. The harness prompt sets a pace in tool calls
and does not ask for short thoughts, so the instruction adds something the prompt lacks.

Different from the null and harmful rows it resembles. E4B "think briefly" (28 / 28 / 25 vs 29) and E4B "prompt
rules compiled into weights" (rank-16 reconstruction, no gain; PRIOR_E4B.md): here the session limit binds in a
quarter of the episodes, and the vector is the 31B model's own per-layer response to the instruction, checked
against the prompted model on held-out calls (table below). The six close-the-thought edits on 31B: no close push,
no depth gate, a different direction (cosine with the logged end-of-thought contrast not measured).

Risks: a quarter of the solved episodes think for more than 800 tokens before their first edit, and that thought may
be where the patch is designed; a mean shift may carry "an instruction is present" without its content; the same
vector is written at prompt positions, where the real instruction changes nothing before its own position.

Dose. beta = 2 is the dose at which the edit moves the held-out logged path as much as the paragraph itself does
(pre-test table): KL from base 0.0072 against 0.0076 for the prompted model, 2.2% against 2.0% of the thought
positions with a changed top token, 10 of the 15 top-token changes of the prompted model reproduced (control at this
dose: 5), and a stream shift about twice the prompted one at a cosine of 0.4 to 0.5, so a component along the
prompted shift of about its size. beta = 1 is closest to the prompted distribution in KL (0.50 of the distance
removed) and did not shorten thoughts on mid-episode calls in served generation. The sister entry `icv-brief-b40` is
twice this dose.

Fitted on (task, unedited arm of batch b001, call index; all stage-1 tasks, none in a stage-2 list):
mewwts__addict.75284f95.func_basic__9sg9rq7f base 3; scitokens__scitokens-193 aa 5; benoitc__gunicorn.bacbf8aa.func_basic__3napqcbj zero 3; rasbt__mlxtend-1087 zero 22; r1chardj0n3s__parse.30da9e4f.func_pm_remove_cond__k4df18dk zero 1; ross-rotordynamics__ross-177 aa 4; lepture__mistune.bf54ef67.combine_module__dalnbilu aa 3; openmdao__openmdao-3315 aa 22; prettytable__prettytable.ca90b055.func_basic__1mmuxqev zero 3; cool-RR__PySnooper.57472b46.lm_rewrite__031xvy9b zero 3; wireservice__csvkit-800 aa 11; life4__textdistance.c3aca916.combine_file__ciwv50c6 base 2; ishepard__pydriller-84 zero 2; zopefoundation__zope.configuration-34 base 26; stingraysoftware__stingray-234 base 29; jsvine__pdfplumber.02ff4313.func_pm_remove_assign__1v9soh1j zero 3; sqlfluff__sqlfluff-5362 zero 24; prometheus__client_python-1134 base 3; Textualize__textual-1352 zero 19; ipython__traitlets-517 aa 28; django-money__django-money.835c1ab8.combine_module__bqggrnha aa 13; pallets__jinja.ada0a9a6.func_basic__2vdao9be zero 27; geographika__mappyfile-146 zero 21; pypa__hatch-1148 aa 13; gcaufield__MonkeyPack-27 base 2; hdmf-dev__hdmf-1270 base 15; tophat__syrupy-768 zero 2; rubik__radon.54b88e58.func_basic__8pbqxg0r zero 8; juju-solutions__charms.reactive-71 zero 32; manicmaniac__sqlalchemy-repr-7 base 1; keis__base58-46 base 3; urbandataanalytics__SwissKnife-12 base 3; pytask-dev__pytask-200 zero 27; conda__conda-6131 aa 31; mewwts__addict.75284f95.combine_file__3fttj8ti zero 1.
Exact recipe
This entry: instruction `I1`, dose beta = 2. Control: the same build with `ctrl`.

base: champion generation 0

No edit had passed when this entry was built; statistics, weights and pre-tests are those of the unedited model.
The entry holds only this change.

size: whole network 0.6713% over 180 matrices, largest matrix 1.6113%

control size: whole network 0.6712% over 180 matrices, largest matrix 1.6113%

The size is mostly the removal of the 60 replaced neurons (old gate and up rows of norm about 1) and is not the
functional dose, which is beta (see "stream shift" below).

Built from the statistics of `code/icv_job.py` (GPU-0 hold 1 of this turn, forward pass `tools/g31.py` only, no
server, no optimiser anywhere; doses above 2 and the controls of doses 2 and 4 were built on CPU by
`code/icv_build.py` and evaluated in hold 2 by `code/icv_job.py --eval-only`, `code/gpu_job6b.sh`) and
rebuilt bit for bit by `build.sh` (`code/icv_build.py` from `code/data/icv_stats.pt`; the sha256 of the rebuilt
`edit/adapter_model.safetensors` equals that of the arm the job built and evaluated, `A20`).

Data. `tools/data/probe_calls_b001_full.jsonl` (one record per logged model call: the prompt the server saw plus the
tokens the agent then wrote; unedited arms `base`, `aa`, `zero` of batch b001). Kept: stage-1 tasks
(`queue/sets/stage1_ids.txt`) that are in no stage-2 list, calls after the first of an episode, at most 4,600 tokens,
a completion of 24 tokens or more that holds a closed thought. One call per task (the middle one by call index).
Tasks sorted by name; every fourth (index % 4 == 1) is held out for the pre-test, the others are fit tasks, shuffled
with seed 6. The 35 fit calls captured before the capture deadline are named at the end of the Hypothesis
section. No task outcome enters any statistic.

Instruction shift. For each fit call the logged token sequence is run twice: as logged, and with the paragraph above
(preceded by a blank line; 56 tokens) inserted at the end of the first user message, before its `<turn|>`.
`resid_post` (layer output after `layer_scalar`) is captured at all 60 layers at the positions of the completion
(at most 300 per call, evenly spaced; with the insertion the same tokens sit 56 positions later). `d_L` = mean
over fit calls of the mean over those positions of (with - without). The part layer L adds itself:
`delta_L = d_L - layer_scalar_L * d_(L-1)`, `d_(-1) = 0`. Size of `d_L` against the stream norm at the same positions,
split-half cosine of `d_L` between even and odd fit calls, and `|delta_L| / |d_L|`, layers [0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 59]:
|d|/|stream| 0.001 0.002 0.002 0.002 0.002 0.005 0.008 0.009 0.008 0.011 0.007; split-half cosine 0.92 0.97 0.96 0.89 0.83 0.87 0.89 0.75 0.73 0.66 0.74;
|delta|/|d| 1.00 0.52 0.38 0.25 0.67 0.56 0.64 0.48 0.29 0.36 0.73.

Always-on neuron (one per layer, replaced outright). The neuron: third smallest mean |activation| in layer L over
the positions sampled in my turn-3 capture (`scratch/job3/net_stats.pt`, 32 stage-1 episodes of b001). Read vector
`w_L`: the mean MLP input (`mlp_in`, after the pre-feedforward norm) over completion positions of the fit calls with
index % 3 != 0 (100 positions per call), scaled so that `w . x` has mean 1 there; a ridge alternative (regress the
constant 1 on `mlp_in`, dual form, lambda = mean squared coordinate) replaces it in a layer where its sd on the
calibration calls (index % 3 == 0) is under 0.7 of the mean vector's (layers that use it: 60 of 60). If `w . x` at the
BOS position exceeds 3 in absolute value, the BOS input is projected out of `w` and `w` is rescaled (layers: 0 of 60).
Gate row = 8 `w` (GELU in its linear range), up row = `w`, so the activation is about 8 (w . x)^2. `gbar_L` = its mean over
completion positions (bf16-rounded rows). Its coefficient of variation on the calibration calls is 0.05 to
0.17 over the 60 layers: about constant, not constant.

Write. Down column of the neuron: `c_L = clipdiv(beta * delta_L, w_post_L) * rmsy_L / (gbar_L * layer_scalar_L)`:
`w_post` = weight of the post-feedforward norm, `rmsy_L` = mean RMS of the raw MLP output at completion positions
of the fit calls, `clipdiv` divides coordinate-wise and zeroes the 5% of coordinates with the smallest |w_post|
(`code/thinklib.py`). With the neuron at `gbar` the stream gets `beta * delta_L` at layer L, to first order (the
neuron's own contribution to the RMS is ignored). `|c_L| gbar_L / (rmsy_L sqrt(5376))`, the write relative to the
RMS of the raw MLP output, is in `edit/edit.json` (`rho`).

Intended against built. Intended: the model behaves at every position as if the paragraph were in the prompt. Built:
a per-layer constant (times the activation factor above) equal to the mean incremental shift on the logged
completions of 35 calls. Not built: the position-dependent part of the instruction's effect, anything the
instruction does through attention patterns, and any correction for later layers responding to the injected shift
of earlier ones (the table "stream shift" shows the result of that).

Control: same neurons, same gate and up rows, and the written vector `beta * delta_L` with the sign of every
coordinate flipped at random (seed 6000 + L) before the same `clipdiv` and scaling: identical per-coordinate
magnitudes, so the same norm in the stream, in the raw MLP output and in |dW|_F, and no relation to the
instruction. It is the right control because it holds the removal of 60 neurons and an always-on write of the same
size in every layer, and differs only in the direction written.

### Pre-test (forward pass, teacher-forced on the logged path; held-out tasks)

`code/icv_job.py` and `code/icv_gen.py` are my own scripts, first used this turn; they were checked by a CPU
plumbing run (`--mock`) and by the bit-for-bit rebuild, not against another tool, so read their numbers with that
in mind. 8 held-out calls is a small sample: on the flip columns the controls (1.1% to 1.3% of the thought positions,
4 to 5 of the 15 prompted flips matched) are not separable from dose 1; KL from base and the stream-shift cosine
separate edit from control at doses 1, 2 and 4, "reproduced" at doses 1 and 2.

`code/icv_job.py`, function `evaluate`: for each held-out call the logged sequence is run unedited (base), unedited
with the paragraph inserted (prompted), and with each adapter and no paragraph; all numbers are at the positions
of the logged completion, against base. "flips" = share of positions with another top token; "close on top inside"
= thought positions where `<channel|>` becomes the top token; "dlogp close" = mean change of log p(`<channel|>`)
where the agent went on / where it closed (and how many closes keep `<channel|>` on top); "reproduced" =
1 - KL(prompted || arm) / KL(prompted || base); "shift cos" = cosine between the arm's and the prompt's change of
the log-probs of base's 50 top tokens; "prompt flips matched" = of the positions where the prompted model's top
token differs from base, those where the arm's top token is the prompted one.

| arm | calls / positions | flips | flips in thought | close on top inside | dlogp close inside | dlogp close at close | tool flips | KL from base | reproduced | shift cos | prompt flips matched |
|---|---|---|---|---|---|---|---|---|---|---|---|
| prompted with the paragraph (the target) | 8 / 1278 | 0.012 | 0.020 | 0 of 460 | -0.07 | -0.192 (7 of 8 kept) | 0 of 8 | 0.008 | - | - | - |
| **this entry** (A20) | 8 / 1278 | 0.013 | 0.022 | 0 of 460 | +0.05 | -0.171 (7 of 8 kept) | 0 of 8 | 0.007 | 0.45 | 0.34 | 10 of 15 |
| its control (A20c) | 8 / 1278 | 0.007 | 0.011 | 0 of 460 | -0.20 | -0.006 (8 of 8 kept) | 0 of 8 | 0.001 | -0.02 | 0.12 | 5 of 15 |
| dose 1 (A10) | 8 / 1278 | 0.008 | 0.009 | 0 of 460 | +0.03 | -0.062 (7 of 8 kept) | 0 of 8 | 0.002 | 0.50 | 0.31 | 7 of 15 |
| dose 1, control (A10c) | 8 / 1278 | 0.007 | 0.011 | 0 of 460 | -0.09 | +0.008 (8 of 8 kept) | 0 of 8 | 0.001 | 0.02 | 0.13 | 4 of 15 |
| dose 3 (A30) | 8 / 1278 | 0.023 | 0.037 | 0 of 460 | +0.19 | -0.294 (6 of 8 kept) | 0 of 8 | 0.016 | -0.09 | 0.34 | 11 of 15 |
| dose 4 (A40) | 8 / 1278 | 0.031 | 0.048 | 0 of 460 | +0.34 | -0.433 (6 of 8 kept) | 0 of 8 | 0.028 | -1.08 | 0.34 | 11 of 15 |
| dose 4, control (A40c) | 8 / 1278 | 0.009 | 0.013 | 0 of 460 | -0.36 | -0.057 (6 of 8 kept) | 0 of 8 | 0.005 | -0.51 | 0.11 | 5 of 15 |

Stream shift (mean over completion positions of `resid_post`, arm minus base, against prompted minus base, same
held-out calls; layers [0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 59]): this entry: cosine 0.25 0.68 0.51 0.46 0.46 0.48 0.54 0.45 0.46 0.42 0.37; norm ratio 3.98 2.04 2.37 2.18 1.87 2.18 1.88 2.27 2.12 1.94 2.48. Control: cosine -0.03 -0.04 0.05 -0.04 0.04 0.05 0.09 0.08 0.13 0.12 0.09; norm ratio 3.95 1.72 1.59 1.14 1.15 1.38 1.39 1.41 1.41 1.30 1.64.

Activation of the always-on neuron over gbar in the edited forward pass (dose 1, held-out calls, 11 layers):
completion positions mean 0.94 to 1.00, sd 0.06 to 0.12, min 0.54, max 1.57; prompt positions
mean 0.76 to 0.92; BOS 0.25 to 3.25.

### Served generation (vLLM through `tools/gpu0_session.sh`, greedy, single turns, no rollout)

(a) `code/icv_gen.py`: 17 mid-episode calls of the held-out tasks (one per task, call indices 2 to 26, prompts of at
most 4,600 tokens, so calls with short thoughts), answers of up to 1,500 tokens. "shorter / longer" = contexts whose
thought is shorter / longer than the unedited model's; "cut" = answer not finished in 1,500 tokens; "loops" =
answers of 600 tokens or more with more than half repeated 8-token sequences.

| arm | thought tokens mean | median | shorter / longer | answer tokens mean | same tool as unedited | well-formed tool call | cut | loops |
|---|---|---|---|---|---|---|---|---|
| unedited | 48 | 30 | - | 139 | - | 16 of 17 | 0 | 0 |
| unedited, paragraph in the prompt | 35 | 25 | 7 / 0 | 116 | 16 of 17 | 16 of 17 | 0 | 0 |
| dose 1 | 71 | 35 | 3 / 3 | 136 | 14 of 17 | 17 of 17 | 0 | 0 |
| dose 2 (**this entry**) | 45 | 30 | 4 / 3 | 124 | 16 of 17 | 17 of 17 | 0 | 0 |
| dose 4 | 41 | 26 | 6 / 5 | 104 | 17 of 17 | 16 of 17 | 0 | 0 |

In the one context with a long unedited thought (264-token answer) the paragraph and doses 2 and 4 make the same
cut (answers of 124, 124 and 119 tokens: the model stops re-reading the HTML and reads the source file), dose 1
does not (267).

(b) `tools/vgen.py --at turn`, dose 1 only: first calls of 16 episodes of the default per-call set (b001; not
separated from the fit tasks), answers up to 1,000 tokens. Thought tokens, unedited: mean 288, median 127. Dose 1:
mean 204, median 129; shorter in 10 contexts, longer in 2; 16 of 16 well-formed tool calls. The same machinery
with the vector of another paragraph ("first edit by your sixth tool call", dose 1): mean 270, 4 shorter, 4 longer,
one thought not closed. Doses 2 and 4 were not run on these contexts.

`interp/toolcall_check.py` on both files: 0 unpaired string delimiters in the 15 to 17 tool calls of every arm
(UNDECIDED: it needs 60 calls); no answer reaches 1,500 tokens, so no loop rate.

`tools/closeprobe.py` (40 logged closes, 126 goes-on points) was run at dose 1 only: change of log p(close) at close
+0.006, at goes-on points +0.077, `<channel|>` on top at 40 of 40 closes. At this entry's dose the quantity exists
only in the table above, on 8 closes: -0.17 (7 of 8 kept; the prompted model: -0.19, 7 of 8).
The tools lane's rule (below -0.05: expect longer ordinary thoughts) flags that value. In the 17 generated
mid-episode turns the thoughts were not longer (table), and the paragraph itself has the same sign.

Not measured: whole episodes; thoughts beyond 1,500 tokens (where the loops of 0020 and 0021 showed); tool-call
syntax on 60 or more calls; generation speed under this adapter.

Checker: `check_adapter.py` passes on `edit/` and `ctrl/` (run by `build.sh`).

Runner facts (2026-11-13 19:51:57 EST): source `wsp31/authors/claude/queue/icv-brief-b20/`; served copy `wsp31/queue/frozen/claude__icv-brief-b20/`; sha256 over files, edit `76a1ee20ff52a9746e80ea00938548822547deda793153e0082e07c0b492d079`, control `160da8249b1d9bab2b200197967c69ae6711acd313943ef74b9711717b944fc4`; both passed the checker.
Prediction (written before the run)
predicted_gain_120: -6 to 6

My median is -1: the measured record of thought-length edits on this model is harmful or null, the pre-tests are
teacher-forced or single turns, and whether the paragraph itself would gain tasks as a prompt is not known (I may
not run rollouts, so the paragraph was never tried as a prompt).

Behaviour that should move if the mechanism holds (edit against the reference arm of the same batch,
`code/arm_stats.py` on the two run directories; reference values are b004 base and aa):
- mean thought tokens per call lower by 10% or more (reference 234 and 240; edit 213 or less);
- share of calls with a thought of 300 tokens or more: 0.20 and 0.21 in the reference, 0.18 or less in the edit;
- median completion tokens per episode lower by 5% or more (reference 6,810 and 7,182);
- episodes at the session limit lower by 2 or more per 102 (if the runner cuts episodes by its token clock,
  `token_clock` in `queue/config.json`, read this as episodes cut by that clock);
- dose order: `icv-brief-b40` moves each of these further than `icv-brief-b20`;
- looping long thoughts (`interp/toolcall_check.py`) not above the unedited range of 0 to 4.3 per 100 episodes,
  unpaired string delimiters below 3% of tool calls;
- the first-edit call index does not have to move.

Control (the same always-on neurons writing a sign-scrambled vector of the same size): none of the above; thought
tokens per call and tokens per episode within the spread of the two reference arms; gain -4 to +3.

The hypothesis is wrong if (a) mean thought tokens per call are not lower than in the reference by at least 5%
(the compiled vector does not act as the instruction in a rollout), or (b) thoughts are shorter by that much and
the session-limit count does not fall (the saved tokens go elsewhere, as for 0015), or (c) thoughts are shorter,
cut-offs fall, and solved tasks do not rise (the long thoughts were doing the work: then the token budget is not
the lever, for any edit of this family). A gain of +3 or more with (a) true would not be this mechanism.
Loops above 10 per 100 episodes or broken tool-call syntax mean the always-on write breaks the model at this
dose, and say nothing about the instruction.
Result
**Stage 1** (batch `x001`, 4 arms on one server, concurrency 4 per arm, finished 2026-11-14 00:41:07 EST; `results/wsp31/batches/x001/`). Reference arm: unedited. Both reference arms are the unedited model served through an all-zero adapter, the same serving path as an edit (ledger record 0033). Box batch: run on rented boxes as 4 shards by task, every arm of a shard on one server; its edits are compared only with its own reference arms (run by hand, adopted).

- unedited: solved 72 of 120, agent errors 46, at session limit 1, at tool-call cap 11, empty patch 21, median tool calls 18.0, mean 281.8 s, median 25.1 completion tokens/s of model time.
- edit: solved 77 of 120, agent errors 32, at session limit 1, at tool-call cap 7, empty patch 13, median tool calls 18.0, mean 266.1 s, median 24.6 completion tokens/s of model time; vs base on 120 common tasks: +5.0 tasks (95% interval -2.0 to +13.0), base-only 5, arm-only 10, exact p 0.3018.
- second unedited arm (identical weights): solved 64 against 72; 13 tasks solved only by the first, 5 only by the second (18 tasks differ; record 0003 measured 11.3 per identical pair); median 24.8 completion tokens/s. The gain used by the rule is against the mean of the two reference arms per task.
- edit against the mean of the two reference arms: +9.0 tasks of 120 (95% interval +2.0 to +16.0).
- Predicted gain -6 to +6; measured +9.0: outside the interval.
- Gain +9.0 is at or above the stage-1 bar (+3): goes to stage 2 with its control.

**Stage 2** (batch `b008`, 4 arms on one server, concurrency 4 per arm, finished 2026-11-14 07:33:26 EST; `results/wsp31/batches/b008/`). Reference arm: unedited. Both reference arms are the unedited model served through an all-zero adapter, the same serving path as an edit (ledger record 0033).

- unedited: solved 114 of 218, agent errors 91, at session limit 90, at tool-call cap 36, empty patch 52, median tool calls 19.0, mean 386.3 s, median 19.0 completion tokens/s of model time.
- edit: solved 123 of 218, agent errors 57, at session limit 57, at tool-call cap 33, empty patch 41, median tool calls 19.0, mean 362.1 s, median 18.8 completion tokens/s of model time; vs base on 218 common tasks: +9.0 tasks (95% interval -1.0 to +20.0), base-only 9, arm-only 18, exact p 0.1221.
- second unedited arm (identical weights): solved 111 against 114; 16 tasks solved only by the first, 13 only by the second (29 tasks differ; record 0003 measured 11.3 per identical pair); median 19.0 completion tokens/s. The gain used by the rule is against the mean of the two reference arms per task.
- edit against the mean of the two reference arms: +10.5 tasks of 218 (95% interval +1.5 to +19.5).
- control: solved 119 of 218, agent errors 84, at session limit 84, at tool-call cap 26, empty patch 46, median tool calls 19.0, mean 387.5 s, median 19.2 completion tokens/s of model time; vs base on 218 common tasks: +5.0 tasks (95% interval -5.0 to +15.0), base-only 12, arm-only 17, exact p 0.4583.
Verdict and notes
null by the fixed rule: total gain +19.5 (needs +10), stage-2 gain +10.5 (needs +4), edit minus control on stage 2 +4.0 (needs +5).
Correction record 0045 — supersedes original counts
corrected solved: base 114 to 113, e0031 123 to 122, c0031 119 to 120, aa 111 to 112 (of 218); verdict changes: none

Written 2026-11-14 09:13:29 EST.

| arm | solved before | solved corrected | gain before | gain corrected |
|---|---|---|---|---|
| `base` | 114 of 218 | 113 of 218 | - | - |
| `e0031` | 123 of 218 | 122 of 218 | +10.5 | +9.5 |
| `c0031` | 119 of 218 | 120 of 218 | +6.5 | +7.5 |
| `aa` | 111 of 218 | 112 of 218 | - | - |

Gain: tasks over the per-task mean of the two reference arms (or the one reference arm), as the rule uses it. "Before" is the batch's own rows rescored with the same scorer (`supp1/score_before.json`; the batch's `score.json` is kept as written).

Per rerun task, before and after, every arm:

| task | `base` | `e0031` | `c0031` | `aa` |
|---|---|---|---|---|
| `pvlib__pvlib-python-509` (b008-supp1) | solved to not solved (empty patch) | solved to not solved (empty patch) | not solved (7.5-minute limit) to solved | not solved (7.5-minute limit) to solved |
| `python-scim__scim2-models-126` (b008-supp1) | not solved to not solved (empty patch) | not solved to not solved | not solved to not solved | not solved to not solved |

Reference arms `base` and `aa` differ on 29 task(s) before and 29 after.

Scored on 218 of the 218 tasks of the final list.

n/a (a correction).

- 0031 (`e0031`): stage-2 gain +10.5 to +9.5; by the fixed rule null before, null corrected. Same verdict; the entry's stage-2 counts are now the corrected ones.