Thought-gated reliability-shrunk Problem Statement spec-audit steering across all 60 MLP layers
When
Recorded date: 2026-11-14
Ledger date only; exact weight-build/change timestamp is not recorded here. Not a live training event.
Exact tensor-change timestamp: unavailable.
Why change it?
Under the doubled `x2` episode budget (20,000 generated tokens, 60 tool calls, 80 turns; batches `x003` and `x004`), forensic inspection of 416 unedited `safe_s1` episodes reveals a structural shift in the dominant bottleneck: 1. **`62.5%` (`90 / 144`) of all unedited failures are premature `submit_patch` (`WRONG_CLEAN`)** with `strict_clean_patch=True`, submitted at a median of **`10,110` generated tokens** (leaving half the 20,000-token budget unused) and **`25.5` tool calls**, failing a median of **`1.0` test** (`FAILED: 1`). Side-by-side patch inspection confirms happy-path-only patches missing explicit Problem Statement boundary/type…
Which part changed?
Thought-gated reliability-shrunk Problem Statement spec-audit steering across all 60 MLP layers
The source describes adapter targets and mathematical changes. No before/after tensor comparison is available in this viewer.
How was it changed?
Compare activations with and without a requirements-and-edge-cases audit instruction. Replace the quietest feedforward neuron in each of 60 layers; modify its gate/up rows and down column. The gate is fitted to activate during reasoning while suppressing changes to tool syntax. Rank-one adapter over 180 matrices.
Did it improve?
null · stage 1: +2.5 of 120 tasks
Evaluation set: wsp31_screen_v2 · Batch: x006. Local screening and historical task scores are not competition scores unless explicitly identified in the source. Different evaluation sets cannot be compared as a gain.
Original evidence
- research/wsp31/ledger/0043-gemini-spec-audit-thought-gate.md
- research/wsp31/paper/tables/leaderboard.md
Literal source excerpts follow. They may contain original predictions and superseded counts; use the measured-outcome summary and corrections above for the current result.
Hypothesis
Under the doubled `x2` episode budget (20,000 generated tokens, 60 tool calls, 80 turns; batches `x003` and `x004`), forensic inspection of 416 unedited `safe_s1` episodes reveals a structural shift in the dominant bottleneck: 1. **`62.5%` (`90 / 144`) of all unedited failures are premature `submit_patch` (`WRONG_CLEAN`)** with `strict_clean_patch=True`, submitted at a median of **`10,110` generated tokens** (leaving half the 20,000-token budget unused) and **`25.5` tool calls**, failing a median of **`1.0` test** (`FAILED: 1`). Side-by-side patch inspection confirms happy-path-only patches missing explicit Problem Statement boundary/type cases (`None`/empty/falsy inputs, container vs scalar types, single vs double quotes, option combinations, sibling/caller code paths). 2. Unconditional brevity steering (`0038 fcv-brief-b20`) turned negative (`-2.0` on `safe_s1`) under `x2` because truncating thoughts rushed verification and increased `wrong_clean` from `23.5` to `31` (`+7.5` wrong patches), whereas adding exactness (`0039 fcv-brief-exact-b20`, `+2.0`) recovered `+4.0` tasks over `0038` despite `94%` of its vector still coming from `"think briefly"` and unshrunk `fcv` subtraction injecting negative split-half noise (`-0.15` to `-0.26`) in upper layers `L36..59`. 3. Architecturally (`ARCH.md`), 50 of 60 layers use 1,024-token sliding windows; only the 10 global-attention layers (`5, 11, 17, 23, 29, 35, 41, 47, 53, 59`) directly attend back to the Problem Statement (`step[1]`, tokens `~2,023..2,506`) in mid/late-episode turns. We hypothesize that replacing 1 near-dead MLP neuron per layer (`0..59`, 180 matrices) with (a) a **non-negative quadratic GeGLU thought-only gate** (`up_row = gate_row / 3.0`, strictly `>= 0` everywhere, active inside `<|channel>thought` and `150x` suppressed on BOS, prompt, `<|"|>` delimiters, and JSON tool arguments) and (b) a **Gemma-4 `layer_scalar`-exact, Spearman-Brown reliability-shrunk incremental stream write** distilled purely from an adversarial Problem Statement spec-and-edge-case audit paragraph (`P_audit`, zero `"think briefly"` text) will steer global-attention queries back to the Problem Statement and elicit targeted `python3 -c` boundary checks before `submit_patch` without breaking tool-call syntax.
Exact recipe
base: champion generation 0
size: whole network 0.9960% over 180 matrices, largest matrix 12.7118%
control size: whole network 0.9961% over 180 matrices, largest matrix 12.7124%
- **Target modules & rank**: `r = 1`, `lora_alpha = 2`, editing `180` MLP matrices (`gate_proj`, `up_proj`, `down_proj` across all 60 layers `0..59`), replacing the single quietest MLP neuron `j_0(l)` per layer (ranked lowest by `max_all |mlp_act| + 10 * mean_all |mlp_act|` across all tokens including BOS on the 32 fit calls).
- **Stage-1 safe data hygiene (zero Stage-2 or sealed-set contact)**: All fitting and pre-testing use only `safe_s1 = (stage1_ids & stage1_ids_v2) - (stage2_ids | stage2_ids_v2 | stage2_ids_240)` (`104` tasks; `78` fit pool, `26` unseen held-out pool), using unedited `x003/base` request logs with zero outcome (`resolved`) labels.
- Pre-edit fit calls (16): ishepard__pydriller-84#c5(read_file), Mimino666__langdetect.a1598f1a.lm_rewrite__dk9wx21q#c5(read_file), conda__conda-6131#c22(run_command), alerta__python-alerta-client-205#c5(run_command), google__yapf-59#c31(run_command), gweis__isodate.17cb25eb.func_basic__ydtogylj#c4(run_command), googleapis__python-api-core-86#c17(read_file), qiskit__qiskit-11272#c10(run_command), pallets__markupsafe.620c06c9.func_basic__fpb7w0tc#c3(read_file), tophat__syrupy-768#c4(read_file), zopefoundation__zope.configuration-34#c11(read_file), janehmueller__python-json-config-47#c4(read_file), lepture__mistune.bf54ef67.combine_module__dalnbilu#c3(read_file), prettytable__prettytable.ca90b055.func_basic__1mmuxqev#c5(run_command), kurtmckee__feedparser.cad965a3.pr_469#c3(read_file), jpadilla__pyjwt-847#c4(read_file)
- Post-edit fit calls (16): pvlib__pvlib-python-834#c8(run_command), r1chardj0n3s__parse.30da9e4f.func_pm_remove_cond__k4df18dk#c6(run_command), h5io__h5io-79#c58(submit_patch), cool-RR__PySnooper.57472b46.lm_rewrite__g0pq8oow#c13(run_command), fgmacedo__python-statemachine-370#c11(submit_patch), pypa__setuptools_scm-299#c24(read_file), ross-rotordynamics__ross-177#c24(submit_patch), joke2k__django-environ-361#c10(run_command), tianocore__edk2-pytool-library-365#c9(submit_patch), django-money__django-money.835c1ab8.lm_rewrite__v5cndhkm#c37(read_file), jsvine__pdfplumber.02ff4313.lm_rewrite__1nhq0adm#c30(run_command), Nikoleta-v3__blackbook-36#c12(submit_patch), Textualize__textual-1352#c9(run_command), copier-org__copier-2646#c29(run_command), mwouts__jupytext-1443#c33(read_file), twisted__twisted-12040#c20(edit_file)
- **Prompted teacher `P_audit` (inserted at end of user message `t106[1]`)**:
`"Before editing or calling submit_patch, re-read every requirement and example in the Problem Statement line by line and audit your diff against edge cases: None/empty/falsy (0, False, "", [], {}) inputs, non-string or container types (int, dict, list, tuple, subclass), single vs double quotes and whitespace, option combinations, and all caller/callee or sibling code paths. Run a targeted python3 -c command testing these boundary cases before calling submit_patch; never submit after only the happy-path test passes."`
- **Closed-form gate (`thought_quad`)**: Dual-Cholesky ridge regression (`4`-fold out-of-task CV across the 32 fit calls for regularization `lambda` and out-of-fold gain calibration) on `mlp_in`, targeting `+3.0` on completion `<|channel>thought` body tokens (weight `0.50`) and `-3.0` on `<channel|>`, `<|"|>` delimiters (`tok 52`), JSON tool arguments, BOS, and prompt tokens (weight `0.50`). Setting `up_row = gate_row / 3.0` yields `act(x) = gelu_tanh(z) * (z / 3.0) >= 0` for all real `z` (eliminating the `-0.17` negative GeLU trough). On 14 unseen `safe_s1` held-out calls (`HIPS__autograd.ac044f0d.lm_rewrite__cetfos0u#c5(pre:read_file), PyCQA__flake8.cf1542ce.lm_rewrite__yh5du3ns#c5(pre:run_command), azure__walinuxagent-1183#c4(pre:read_file), brazilian-utils__brutils-python-114#c5(pre:read_file), cookiecutter__cookiecutter.b4451231.lm_rewrite__byoy8vaf#c4(pre:read_file), django-money__django-money.835c1ab8.combine_module__bqggrnha#c4(pre:run_command), gammapy__gammapy-5300#c3(pre:run_command), hhursev__recipe-scrapers-1187#c15(post:read_file), jd__tenacity.0d40e76f.combine_module__jua0momz#c57(post:read_file), jsvine__pdfplumber.02ff4313.func_pm_remove_assign__1v9soh1j#c18(post:submit_patch), keis__base58-46#c11(post:submit_patch), lepture__python-livereload-198#c14(post:submit_patch), litestar-org__polyfactory-383#c48(post:submit_patch), mc706__changelog-cli-34#c8(post:run_command)`), `thought_quad` achieves **unseen ROC-AUC `0.982` (`L5..59`)** separating thought from tool syntax/delimiters (`mean act = 3.47` on thought vs `0.023` on `<|"|>`/tool-args, `150x` ON/OFF suppression).
- **Closed-form write (`solve_stream_columns`, `dose = 20.0`, `gamma = 0.9`, `eta_max = 0.65`)**:
1. Accounts for Gemma-4 `layer_scalar` `s_l` (`0.032..0.992`) multiplying the entire post-MLP stream: incremental innovation `delta_l = mu_l / s_l - gamma * mu_{l-1}`.
2. Applies split-half (`16` vs `16` fit tasks) Spearman-Brown shrinkage `rho_l = 2*max(0, cos_ab) / (1 + max(0, cos_ab))` (`rho` by 10-layer bands: `[0.349, 0.513, 0.379, 0.048, 0.095, 0.194]`) to `delta_l` so noisy upper-layer components shrink toward `0` rather than subtracting earlier mid-layer steering.
3. Maps stream shift through smooth Tikhonov `post_feedforward_layernorm` inverse `w_post / (w_post^2 + tau_l^2)` (`tau_l = q_0.20(|w_post|)`) with RMSNorm denominator compensation `1/sqrt(1 + eta_l^2)` (`max_eta = 0.291` at `dose = 20.0`, preserving `96.0%` of native `mlp_out`) and carry-forward across small-`w_post` global layers (`L5`, `L11`). Orthogonalizes `d_edit` against `col_base` (`1/5376` dims).
- **Matched control (`ctrl/`)**: Built by the exact same `t9_build.py` code path (`seed = 20260901`): identical neuron indices `j_0(l)` and identical `gate_proj` and `up_proj` rows; `down_proj` column applies the identical Tikhonov `w_post` converter to a random Gaussian stream vector orthogonal to `g_want`, orthogonalized against `col_base` and `d_edit`, and scaled so BOTH `||d_ctrl||_2 == ||d_edit||_2` AND `||d_ctrl - col_base||_2 == ||d_edit - col_base||_2` (`edit` vs `ctrl` whole-network norm `0.9960%` vs `0.9961%`).
- **Deterministic CPU rebuild (`< 5s`, verified bit-for-bit SHA256)**:
- `stats`: `code/data/t9_stats.pt` (`sha256: e2805e49bdf6f7ee...`)
- `edit`: `OK edit: 180 modules changed (of 180 present), r=1, max |dW|_F/|W|_F 0.1271 at 11.down_proj` (`sha256: 0429bc5e74021e48...`)
- `ctrl`: `OK ctrl: 180 modules changed (of 180 present), r=1, max |dW|_F/|W|_F 0.1271 at 11.down_proj` (`sha256: 483ea1ddd1a2d329...`)
- **Pre-test diagnostics on unseen `safe_s1` episodes** *(note: per `TOOLS.md`, probe verdict words/thresholds are not calibrated to rollout task gains)*:
- Offline exact forward pass (`t9_hold1.py`, 14 unseen calls): positive full-vocab thought log-prob shift alignment with true `P_audit` (`cos = +0.237`), `0 / 124` top-1 flips on `<|"|>` string delimiters (`tok 52`) and JSON tool arguments (`KL_syn` `7x` lower than `KL_th`).
- Served `fastprobe.py` on `held26_ep.jsonl` (port 18420, 455 decision points of 20 unseen `safe_s1` traces):
- `edit`: `ACTIVE`, `mean KL = 0.00171` (`max 0.0527`; `turn 0.00012`, `step 0.00293`, `think 0.00092`, `tool 0.00152`), `0.0%` protocol-boundary top-1 flips (`P(<|channel>) 0.998->0.998` at `turn`, `P(<|tool_call>) 0.962->0.961` at `think`).
- `ctrl`: `ACTIVE`, `mean KL = 0.00170` (`max 0.0721`; `turn 0.00010`, `step 0.00267`, `think 0.00168`, `tool 0.00101`), `0.0%` protocol-boundary top-1 flips.
Runner facts (2026-11-14 08:59:09 EST): source `wsp31/authors/gemini/queue/spec-audit-thought-gate/`; served copy `wsp31/queue/frozen/gemini__spec-audit-thought-gate/`; sha256 over files, edit `bc6f446d2110abbfc1eaa0349e2b3e4440bc9b5f28345878a938065a4ca0a663`, control `c58c09d8a0fde5e5d48941b245bec8682791dc2c30e504d977203f564029cd37`; both passed the checker.Prediction (written before the run)
predicted_gain_120: 2.0 to 7.0 - **Expected behavioral mechanism (`x2` 20,000-token budget)**: 1. Fewer premature happy-path `submit_patch` exits: `wrong_clean` unsolved count drops by `3 to 8` tasks relative to zero-adapter `base/aa` as post-edit thoughts audit Problem Statement edge cases and invoke `run_command` (`python3 -c`) or follow-up `edit_file` before submitting. 2. Preserved budget & syntax safety: because `P_audit` contains zero brevity compression and `thought_quad` suppresses `<|"|>`/tool-arg activations by `150x` (`0/124` held-out delimiter/arg flips), unpaired string-delimiter rate stays `< 2.5%` (`NOT BROKEN`) and `cutoff` count stays within `+-3` of `base/aa`. 3. Matched control `ctrl/` scores near zero (`-2.5 to +2.0` vs `base/aa`), with `edit` beating `ctrl` by `>= +3.5` tasks. - **Falsification**: Falsified if `wrong_clean` fails to decrease relative to `base/aa`, if `cutoff` rises by `> 4` tasks from over-auditing, if `ctrl/` matches `edit/` within `1.0` task, or if Stage-1 gain is `< +2.0`.
Result
**Stage 1** (batch `x006`, 4 arms on one server, concurrency 4 per arm, finished 2026-11-14 11:26:41 EST; `results/wsp31/batches/x006/`). Reference arm: unedited. Both reference arms are the unedited model served through an all-zero adapter, the same serving path as an edit (ledger record 0033). Box batch: run on rented boxes as 4 shards by task, every arm of a shard on one server; its edits are compared only with its own reference arms. Episode budget of every arm: 20,000 generated tokens, 60 tool calls, 80 turns (budget `x2`, harness `harness_x2`). - unedited: solved 78 of 120, agent errors 24, at session limit 24, at tool-call cap 4, empty patch 5, median tool calls 23.0, mean 405.7 s, median 24.0 completion tokens/s of model time. - edit: solved 79 of 120, agent errors 20, at session limit 20, at tool-call cap 1, empty patch 5, median tool calls 20.5, mean 418.3 s, median 24.0 completion tokens/s of model time; vs base on 120 common tasks: +1.0 tasks (95% interval -5.0 to +7.0), base-only 4, arm-only 5, exact p 1.0. - second unedited arm (identical weights): solved 75 against 78; 10 tasks solved only by the first, 7 only by the second (17 tasks differ; record 0003 measured 11.3 per identical pair); median 23.7 completion tokens/s. The gain used by the rule is against the mean of the two reference arms per task. - edit against the mean of the two reference arms: +2.5 tasks of 120 (95% interval -1.5 to +6.5). - Predicted gain +2 to +7; measured +2.5: inside the interval.
Verdict and notes
null by the fixed rule: stage-1 gain +2.5, bar +3, harmful at -6 or below.