GEMMA DEVELOPER AGENTSAVED EVIDENCE
← Run archive
RUN DETAIL

31B real rollouts · 480 (exact stack, 32 lanes, 13-Nov)

60 tasks, 480 rollouts. 261 solved (54%). Open any rollout to read it turn by turn: the model's reasoning, each tool call and its result, and every generated token.

261solved
219not solved
480of 480 rollouts readable
Run settings and provenance
{
  "adapter": "rank-16 LoRA loaded with all-zero B weights, so it adds nothing (base model)",
  "evaluation_set": "60 SWE tasks x 8 rollouts (rollout31b-real test kit)",
  "model": "Gemma 4 31B it, QAT W4A16 (gemma-4-31b-it-qat-w4a16-ct)",
  "name": "31B real rollouts · 480 (exact stack, 32 lanes, 13-Nov)",
  "notes": "The first box was taken back by Spot after 207 rollouts; the restart reran all 480. Only the restart's rows are shown (last row per run_key).",
  "result_kind": "local rollout bench (not a Kaggle score)",
  "row_selection": "last row per run_key",
  "rows_kept": 480,
  "settings": {
    "faucet_job": "rollout31b-480-1 (4e4aea5c876851a0), attempt 3",
    "gpu": "1x NVIDIA RTX PRO 6000 Blackwell (faucet box g4-standard-48, Spot)",
    "lanes": 32,
    "session_cap_minutes": 7.5,
    "stack": "exact (lossless) rollout stack, vLLM 0.19.1, torch 2.10.0+cu128",
    "tool_call_budget": 30
  },
  "source": "gs://cellensml-training-jobs/jobs/son/4e4aea5c876851a0/s0/a3/working/r31/rollouts_X",
  "source_rows": 687,
  "source_sha256": "03479861121e195ca3eb79ae47872470b8bfc369ef52cb0bb26bc0b56feb583d",
  "source_timestamp": "2026-11-13 22:19 to 2026-11-14 00:07 New York time (the complete second attempt)"
}

r0 solved r0 not solved r0 outcome unknown r0 evidence not uploaded yet

TaskRepositorySolvedRollouts
2gis__k8s-handle-1202gis/k8s-handle0 / 8r0r1r2r3r4r5r6r7
amaranth-lang__amaranth-923amaranth-lang/amaranth0 / 8r0r1r2r3r4r5r6r7
apiflask__apiflask-632apiflask/apiflask7 / 8r0r1r2r3r4r5r6r7
asdf-format__asdf-1853asdf-format/asdf6 / 8r0r1r2r3r4r5r6r7
astanin__python-tabulate-268astanin/python-tabulate7 / 8r0r1r2r3r4r5r6r7
Aunsiels__pyformlang-32Aunsiels/pyformlang6 / 8r0r1r2r3r4r5r6r7
azure__walinuxagent-547Azure/WALinuxAgent8 / 8r0r1r2r3r4r5r6r7
backblaze__b2_command_line_tool-488Backblaze/B2_Command_Line_Tool1 / 8r0r1r2r3r4r5r6r7
bokeh__bokeh-13289bokeh/bokeh8 / 8r0r1r2r3r4r5r6r7
boto__botocore-830boto/botocore2 / 8r0r1r2r3r4r5r6r7
caikit__caikit-233caikit/caikit0 / 8r0r1r2r3r4r5r6r7
callowayproject__bump-my-version-180callowayproject/bump-my-version3 / 8r0r1r2r3r4r5r6r7
celery__celery-9465celery/celery1 / 8r0r1r2r3r4r5r6r7
coady__multimethod-26coady/multimethod1 / 8r0r1r2r3r4r5r6r7
cs-si__eodag-961CS-SI/eodag1 / 8r0r1r2r3r4r5r6r7
dask__dask-10521dask/dask8 / 8r0r1r2r3r4r5r6r7
docling-project__docling-2983docling-project/docling7 / 8r0r1r2r3r4r5r6r7
ecmwf__earthkit-data-111ecmwf/earthkit-data8 / 8r0r1r2r3r4r5r6r7
enthought__traits-541enthought/traits3 / 8r0r1r2r3r4r5r6r7
facebookresearch__hydra-273facebookresearch/hydra0 / 8r0r1r2r3r4r5r6r7
fox-it__dissect.cstruct-66fox-it/dissect.cstruct3 / 8r0r1r2r3r4r5r6r7
gammapy__gammapy-5674gammapy/gammapy1 / 8r0r1r2r3r4r5r6r7
glotzerlab__signac-185glotzerlab/signac5 / 8r0r1r2r3r4r5r6r7
h5py__h5py-2577h5py/h5py1 / 8r0r1r2r3r4r5r6r7
hdmf-dev__hdmf-1270hdmf-dev/hdmf2 / 8r0r1r2r3r4r5r6r7
hgrecco__pint-1111hgrecco/pint7 / 8r0r1r2r3r4r5r6r7
iterative__dvc-8904iterative/dvc6 / 8r0r1r2r3r4r5r6r7
jazzband__tablib-613jazzband/tablib1 / 8r0r1r2r3r4r5r6r7
kinegratii__borax-23kinegratii/borax8 / 8r0r1r2r3r4r5r6r7
M0r13n__pyais-137M0r13n/pyais3 / 8r0r1r2r3r4r5r6r7
M3t0r__tpl-12M3t0r/tpl6 / 8r0r1r2r3r4r5r6r7
matthewwithanm__python-markdownify-202matthewwithanm/python-markdownify8 / 8r0r1r2r3r4r5r6r7
mirumee__ariadne-24mirumee/ariadne8 / 8r0r1r2r3r4r5r6r7
modelcontextprotocol__python-sdk-1864modelcontextprotocol/python-sdk8 / 8r0r1r2r3r4r5r6r7
narwhals-dev__narwhals-2864narwhals-dev/narwhals8 / 8r0r1r2r3r4r5r6r7
ncclient__ncclient-418ncclient/ncclient6 / 8r0r1r2r3r4r5r6r7
neurodatawithoutborders__pynwb-1669NeurodataWithoutBorders/pynwb7 / 8r0r1r2r3r4r5r6r7
pallets__werkzeug-2194pallets/werkzeug6 / 8r0r1r2r3r4r5r6r7
pylons__waitress-249Pylons/waitress1 / 8r0r1r2r3r4r5r6r7
pymodbus-dev__pymodbus-1923pymodbus-dev/pymodbus0 / 8r0r1r2r3r4r5r6r7
pyomo__pyomo-3017Pyomo/pyomo3 / 8r0r1r2r3r4r5r6r7
pyqtgraph__pyqtgraph-3278pyqtgraph/pyqtgraph8 / 8r0r1r2r3r4r5r6r7
python-tap__tappy-112python-tap/tappy8 / 8r0r1r2r3r4r5r6r7
qiskit__qiskit-12527Qiskit/qiskit1 / 8r0r1r2r3r4r5r6r7
qiskit__qiskit-terra-5833Qiskit/qiskit-terra0 / 8r0r1r2r3r4r5r6r7
reanahub__reana-client-701reanahub/reana-client8 / 8r0r1r2r3r4r5r6r7
reata__sqllineage-224reata/sqllineage5 / 8r0r1r2r3r4r5r6r7
researchobject__ro-crate-py-102ResearchObject/ro-crate-py7 / 8r0r1r2r3r4r5r6r7
rigetticomputing__pyquil-115rigetticomputing/pyquil8 / 8r0r1r2r3r4r5r6r7
samuelcolvin__pydantic-3177samuelcolvin/pydantic3 / 8r0r1r2r3r4r5r6r7
scrapy__w3lib-202scrapy/w3lib8 / 8r0r1r2r3r4r5r6r7
sebrut__pygrocy-120SebRut/pygrocy8 / 8r0r1r2r3r4r5r6r7
sfdo-tooling__cumulusci-3624SFDO-Tooling/CumulusCI3 / 8r0r1r2r3r4r5r6r7
Stratoscale__skipper-164Stratoscale/skipper5 / 8r0r1r2r3r4r5r6r7
ucfopen__canvasapi-716ucfopen/canvasapi0 / 8r0r1r2r3r4r5r6r7
useblocks__sphinx-needs-1102useblocks/sphinx-needs1 / 8r0r1r2r3r4r5r6r7
xknx__xknx-1227XKNX/xknx4 / 8r0r1r2r3r4r5r6r7
xlsform__pyxform-489XLSForm/pyxform0 / 8r0r1r2r3r4r5r6r7
zalando__patroni-1991zalando/patroni8 / 8r0r1r2r3r4r5r6r7
zopefoundation__DateTime-62zopefoundation/DateTime1 / 8r0r1r2r3r4r5r6r7