GEMMA DEVELOPER AGENTSAVED EVIDENCE

← Weight change

Direct mathematical edit / wsp31-0050

Instruction pair vector, 'be exact about what the issue asks' part written three times in layers 6 to 24, think-briefly part at dose 2

Gemma 4 31B, frozen four-bit baseAuthor: claudeState: runningVerdict: -

When

Recorded date: 2026-11-14

Ledger date only; exact weight-build/change timestamp is not recorded here. Not a live training event.

Exact tensor-change timestamp: unavailable.

Why change it?

At 20,000 tokens the failures are wrong patches, and the one arm that did not add wrong patches while it cut tokens was the one that carried the exactness paragraph. Batch x004 (120 stage-1 tasks, budget `x2`; `code/x_stats.py`): | arm | solved | finished with a wrong patch | out of budget with a wrong patch | empty patch | median output tokens per episode | |---|---|---|---|---|---| | unedited (two arms) | 78, 76 | 27, 28 | 8, 8 | 7, 8 | 9990, 9021 | | 0038 `fcv-brief-b20` (think briefly, dose 2) | 75 | 34 | 6 | 5 | 6425 | | 0039 `fcv-brief-exact-b20` (think briefly + be exact, dose 2) | 79 | 27 | 11 | 3 | 6838 | The two entries share…

Which part changed?

Instruction pair vector, 'be exact about what the issue asks' part written three times in layers 6 to 24, think-briefly part at dose 2

The source describes adapter targets and mathematical changes. No before/after tensor comparison is available in this viewer.

How was it changed?

Computed directly by author-written code and packaged as a low-rank adapter over the frozen base; not gradient-trained. base: champion generation 0 No edit had passed when this entry was built; statistics, weights and pre-tests are those of the unedited model. size: whole network 0.6819% over 180 matrices, largest matrix 2.0319% control size: whole network 0.6820% over 180 matrices, largest matrix 2.0360% The size is mostly the removal of the 60 replaced neurons (old gate and up rows of norm about 1). Rank 1 per matrix, 180 matrices (gate, up and down of the MLP in all 60 layers). Write of layer L (`code/x3_build.py`, CPU, from `code/data/fcv2_stats.pt`): G_L = self-consistent write of the pair I1 + I5 at dose 2 (delta["G20"]; entry 0039) F_L =…

Did it improve?

- · No task-level result recorded in this snapshot.

Pending or parked proposal; not established as an applied or promoted update.

Evaluation set: wsp31_screen_v2 · Batch: x008. Local screening and historical task scores are not competition scores unless explicitly identified in the source. Different evaluation sets cannot be compared as a gain.

Original evidence

Literal source excerpts follow. They may contain original predictions and superseded counts; use the measured-outcome summary and corrections above for the current result.

Hypothesis
At 20,000 tokens the failures are wrong patches, and the one arm that did not add wrong patches while it cut tokens
was the one that carried the exactness paragraph. Batch x004 (120 stage-1 tasks, budget `x2`; `code/x_stats.py`):

| arm | solved | finished with a wrong patch | out of budget with a wrong patch | empty patch | median output tokens per episode |
|---|---|---|---|---|---|
| unedited (two arms) | 78, 76 | 27, 28 | 8, 8 | 7, 8 | 9990, 9021 |
| 0038 `fcv-brief-b20` (think briefly, dose 2) | 75 | 34 | 6 | 5 | 6425 |
| 0039 `fcv-brief-exact-b20` (think briefly + be exact, dose 2) | 79 | 27 | 11 | 3 | 6838 |

The two entries share neurons, compile, dose and batch; they differ by the mean shift that the second paragraph adds:

> (I5) Important: be exact about what the issue asks. In every thought keep in view the precise behaviour the issue describes, with each example, input variant and error message it mentions, and make your change fix the general cause so that all of them work, not only the case that was reported.

0039 minus 0038 is +4 solved and 7 fewer finished-wrong episodes: one batch, inside the noise of two arms (identical
weights differ on about 12 tasks). In 0039 that part was weak: the pair reproduced 0.40 of the pair prompt on held-out
calls against 0.31 for the brevity paragraph alone, and the part I5 adds to the mean shift is reproducible between
halves of the fit calls only in layers 6 to 24 (split-half cosine 0.54 to 0.89 there; -0.26 to 0.26 from layer 31 up).

This entry is the dose step that tells signal from noise: the same pair vector, with the I5 part written **three
times** where it is reproducible (layers 6 to 24) and unchanged elsewhere. If I5's shift is what removed the wrong
patches, three times as much of it should take the finished-wrong count below the unedited model's; if 0039 minus
0038 was noise, this entry scores like 0038 or 0039 and the constant-vector line for quality instructions is closed.

Different from the null rows it resembles. 0038 (-2.0) and 0039 (+2.0) are the two ends this entry extends; it is a
dose ladder on the one component that separated them, not a third copy. 0035 / 0036 (gpt, source-grounding contrast,
-2.5 and -3.5) carried another instruction with no brevity part. 0043 / 0044 (gemini, audit paragraph, gated to
thoughts; not scored when this was written) use another instruction and a gate; this entry is always-on.

Risks. (1) Three times the I5 part moves the stream in layers 12 to 24 about 5 to 7 times as far as the pair prompt does
there (0039: 2 to 3 times) at a lower cosine with the prompted shift (0.30 to 0.48 against 0.49 to 0.59): this is
outside what a prompt does, by design, and may cost coherence that single-call pre-tests do not see. (2) I have no
reference prompted with I5 alone, so "more of I5" is the arithmetic of the vectors, not a measured match to a
three-times-stronger instruction. (3) Nobody has measured whether I5 as a prompt gains tasks.

Fitted on (task, unedited arm of batch b001, call index; all stage-1 tasks, none in a stage-2 list; the 35
calls of turn 6, no outcome labels): mewwts__addict.75284f95.func_basic__9sg9rq7f base 3; scitokens__scitokens-193 aa 5; benoitc__gunicorn.bacbf8aa.func_basic__3napqcbj zero 3; rasbt__mlxtend-1087 zero 22; r1chardj0n3s__parse.30da9e4f.func_pm_remove_cond__k4df18dk zero 1; ross-rotordynamics__ross-177 aa 4; lepture__mistune.bf54ef67.combine_module__dalnbilu aa 3; openmdao__openmdao-3315 aa 22; prettytable__prettytable.ca90b055.func_basic__1mmuxqev zero 3; cool-RR__PySnooper.57472b46.lm_rewrite__031xvy9b zero 3; wireservice__csvkit-800 aa 11; life4__textdistance.c3aca916.combine_file__ciwv50c6 base 2; ishepard__pydriller-84 zero 2; zopefoundation__zope.configuration-34 base 26; stingraysoftware__stingray-234 base 29; jsvine__pdfplumber.02ff4313.func_pm_remove_assign__1v9soh1j zero 3; sqlfluff__sqlfluff-5362 zero 24; prometheus__client_python-1134 base 3; Textualize__textual-1352 zero 19; ipython__traitlets-517 aa 28; django-money__django-money.835c1ab8.combine_module__bqggrnha aa 13; pallets__jinja.ada0a9a6.func_basic__2vdao9be zero 27; geographika__mappyfile-146 zero 21; pypa__hatch-1148 aa 13; gcaufield__MonkeyPack-27 base 2; hdmf-dev__hdmf-1270 base 15; tophat__syrupy-768 zero 2; rubik__radon.54b88e58.func_basic__8pbqxg0r zero 8; juju-solutions__charms.reactive-71 zero 32; manicmaniac__sqlalchemy-repr-7 base 1; keis__base58-46 base 3; urbandataanalytics__SwissKnife-12 base 3; pytask-dev__pytask-200 zero 27; conda__conda-6131 aa 31; mewwts__addict.75284f95.combine_file__3fttj8ti zero 1.
Exact recipe
base: champion generation 0

No edit had passed when this entry was built; statistics, weights and pre-tests are those of the unedited model.

size: whole network 0.6819% over 180 matrices, largest matrix 2.0319%

control size: whole network 0.6820% over 180 matrices, largest matrix 2.0360%

The size is mostly the removal of the 60 replaced neurons (old gate and up rows of norm about 1). Rank 1 per matrix,
180 matrices (gate, up and down of the MLP in all 60 layers).

Write of layer L (`code/x3_build.py`, CPU, from `code/data/fcv2_stats.pt`):

    G_L = self-consistent write of the pair I1 + I5 at dose 2   (delta["G20"]; entry 0039)
    F_L = self-consistent write of I1 alone at dose 2           (delta["F20"]; entry 0038)
    H_L = G_L + 2 (G_L - F_L)  for 6 <= L <= 24;   H_L = G_L  otherwise

Size of H_L over size of G_L at layers 6, 12, 18, 24: 1.58, 2.99, 2.13, 1.71. The always-on neuron of every layer is
that of 0038 and 0039 (fourth-quietest neuron of the layer, gate row 8 w, up row w, w fitted by ridge so that
w . x is 1 on completion positions; `code/fcv_job.py`); its down column writes H_L through the norm after the MLP
(`icv_job.build_arm`: H_L / w_post, zero where |w_post| is below its 5% quantile, times the MLP output RMS over the
neuron's mean activation and the layer scalar). Size of the write over the MLP output: 0.006 to 0.106, median 0.018.
"Self-consistent" (turn 8): a hooked forward pass in which every layer adds only what its mean shift over the
completion positions still lacks of dose times the prompted mean shift. I1 is the paragraph of 0031: "Important: keep
your thinking very short. Before each tool call, think in at most three short sentences: what you just learned and
what you will do next. Never write long plans, never draft or rewrite code in your thoughts, and never go over the
same point twice." Both paragraphs were inserted at the end of the first user message.

Intended against built: I intended "the pair with I5 at three times its strength". Built: the difference of two
self-consistent compiles, added twice in 19 layers. Layers above 24 still write G_L, which was compiled for a stream
that carries the pair at dose 2, so the extra part propagates upward uncorrected.

Control: the same code path with the sign of every coordinate of the written vector flipped at random (seed 6000 + L):
identical neurons, gate and up rows, identical per-coordinate magnitudes of the write, so identical size. It is an
always-on write of the same size in a direction unrelated to the paragraphs, the right control for "the direction
matters", as in 0031 to 0039.

Rebuild: `build.sh` (CPU, about a minute). The sha256 of `edit/` and `ctrl/` equal those of the arms `H23` and `H23c`
that the GPU job evaluated (`code/fin11a.sh`). Checker: `OK edit: 180 modules changed (of 180 present), r=1, max
|dW|_F/|W|_F 0.02032 at 6.down_proj`; `OK ctrl: ... 0.02036 at 6.down_proj`.

Pre-tests (GPU 0, forward pass `tools/g31.py`, no server; `code/mk_job.py`, `icv_job.evaluate`): teacher-forced on 8
held-out calls of 8 stage-1 tasks not among the fit calls, 1,278 positions, against the model prompted with the pair.

| arm | KL(base to arm) | top token changed, all / in thoughts | share of the pair prompt's effect reproduced | of the prompt's 24 top-token changes matched | tool-name positions changed | log p(close) inside thoughts / at the 8 closes |
|---|---|---|---|---|---|---|
| prompted with the pair | 0.0077 | 1.9% / 3.7% | (target) | 24 | 0 of 8 | -0.006 / -0.20 |
| this entry (H23) | 0.0066 | 1.6% / 2.8% | 0.37 | 13 | 0 of 8 | -0.213 / -0.067 |
| 0039 (G20) | 0.0060 | 1.5% / 2.6% | 0.40 | 13 | 0 of 8 | -0.198 / -0.106 |
| control (H23c) | 0.0016 | 1.1% / 1.5% | -0.15 | 8 | 0 of 8 | -0.090 / -0.007 |

About 20 top-token changes per 1,278 logged tokens, so of the order of 100 per episode: the edit acts under greedy
decoding. It reproduces the pair prompt no better than 0039 (0.37 against 0.40), as expected of a vector pushed away
from the pair along one of its parts; stream shift over prompted shift at layers 12, 18, 24: 5.0, 7.0, 3.2 (0039:
2.2, 3.0, 1.5), cosine 0.45, 0.48, 0.30 (0039: 0.59, 0.56, 0.49); layers 30 to 59: ratio 1.2 to 1.8, cosine 0.43 to 0.58.

Thought ends, `tools/closeprobe.py` (40 logged closes, exact contexts; flag below -0.05):

- this entry (H23): change of log p(close) at close: mean +0.032, raised at 0.850 of 40 positions; change of log p(close) at cont: mean -0.040, raised at 0.405 of 126 positions

Served single turns (`code/icv_gen.py` under `gpu0_session.sh`, port 18400, no tool executed, no rollout; mid-episode
calls of held-out stage-1 tasks, 1,500 tokens at most), and `interp/toolcall_check.py` on them:

| arm | thought tokens mean | median | shorter / longer than unedited | answer tokens mean | same tool as unedited | well-formed tool call | cut | loops |
|---|---|---|---|---|---|---|---|---|
| unedited | 47 | 30 | - | 139 | - | 17 of 17 | 0 | 0 |
| unedited, paragraph I1 alone in the prompt | 35 | 25 | 7 / 0 | 116 | 16 of 17 | 16 of 17 | 0 | 0 |
| Q20 (`mk-pair-global-b20`) | 35 | 30 | 3 / 4 | 128 | 17 of 17 | 17 of 17 | 0 | 0 |
| H23 (`fcv-brief-exact3-b20`) | 44 | 30 | 5 / 4 | 124 | 16 of 17 | 16 of 17 | 0 | 0 |
| Q20c (control of `mk-pair-global-b20`) | 45 | 30 | 1 / 1 | 139 | 17 of 17 | 17 of 17 | 0 | 0 |

17 contexts (calls 2, 17, 4, 17, 9, 3, 12, 12, 21, 26, 2, 13, 12, 17, 4, 26, 15).

`toolcall_check.py`: reference: 0 of 17 tool calls = 0.0%; prompt_I1: 0 of 17 tool calls = 0.0%; Q20: 0 of 17 tool calls = 0.0%; H23: 0 of 17 tool calls = 0.0%; Q20c: 0 of 17 tool calls = 0.0% (unedited rollouts 0.7% to 1.1%; too few calls for the tool's verdict, which needs 60).

In one turn (`pypa__build-229`, call 26) this entry calls `submit_patch` where the unedited model runs a command; the
unedited model with I1 in the prompt does the same there (the generation script counts a `submit_patch` as not
well-formed in both; `toolcall_check.py` counts 17 complete tool calls for every arm). Thoughts are shorter than the
unedited model's in 5 turns and longer in 4: in single turns the brevity is weaker than the prompt's (7 and 0).

Runner facts (2026-11-14 10:17:04 EST): source `wsp31/authors/claude/queue/fcv-brief-exact3-b20/`; served copy `wsp31/queue/frozen/claude__fcv-brief-exact3-b20/`; sha256 over files, edit `3df61c3d315165f04b042bdca5081d222c61544d67e951deb23219c4bae0ec8c`, control `740efbc09d6acee6b276121fbd0fdb4a9eef3b01fa19c00500eb1de97334dd6d`; both passed the checker.
Prediction (written before the run)
predicted_gain_120: -5 to 7

Point estimate +1. Behaviour, against the unedited arms of the same batch (budget `x2`):
- median output tokens per episode 6,000 to 7,500 against about 9,000 to 10,000 (as 0038 and 0039: the brevity part is unchanged);
- out of budget in 10 to 16 episodes against 18 to 21; empty patch 3 to 6 against 7 to 8;
- finished with a wrong patch: 20 to 26 if I5's part does what 0039 minus 0038 suggests (unedited 27, 28; 0038 34;
  0039 27). This is the number the entry is for.
- median tool calls about 19 against 21 to 22.

Control: within -4 to +4 of the reference arms, no change in tokens per episode beyond 10%, finished-wrong 25 to 30.

The hypothesis is wrong if the finished-wrong count is 30 or more (three times the I5 part did not even hold 0039's
level) or the gain is at or below 0038's -2: then 0039 minus 0038 was noise, a constant vector does not carry a
quality instruction, and I stop varying constants. It is unproven if the gain is 0 to +3 with finished-wrong 25 to 29.
A gain of +5 or more with finished-wrong at 24 or below would be a dose response and worth a stage 2.
A drop in solved by 6 or more with more tool calls or longer thoughts means the mid-layer overshoot (risk 1) broke
ordinary behaviour.
Result
Pending.
Verdict and notes
Pending.