RUN DETAIL
31B real rollouts · 480 (exact stack, 32 lanes, 13-Nov)
60 tasks, 480 rollouts. 261 solved (54%). Open any rollout to read it turn by turn: the model's reasoning, each tool call and its result, and every generated token.
261solved
219not solved
480of 480 rollouts readable
Run settings and provenance
{
"adapter": "rank-16 LoRA loaded with all-zero B weights, so it adds nothing (base model)",
"evaluation_set": "60 SWE tasks x 8 rollouts (rollout31b-real test kit)",
"model": "Gemma 4 31B it, QAT W4A16 (gemma-4-31b-it-qat-w4a16-ct)",
"name": "31B real rollouts · 480 (exact stack, 32 lanes, 13-Nov)",
"notes": "The first box was taken back by Spot after 207 rollouts; the restart reran all 480. Only the restart's rows are shown (last row per run_key).",
"result_kind": "local rollout bench (not a Kaggle score)",
"row_selection": "last row per run_key",
"rows_kept": 480,
"settings": {
"faucet_job": "rollout31b-480-1 (4e4aea5c876851a0), attempt 3",
"gpu": "1x NVIDIA RTX PRO 6000 Blackwell (faucet box g4-standard-48, Spot)",
"lanes": 32,
"session_cap_minutes": 7.5,
"stack": "exact (lossless) rollout stack, vLLM 0.19.1, torch 2.10.0+cu128",
"tool_call_budget": 30
},
"source": "gs://cellensml-training-jobs/jobs/son/4e4aea5c876851a0/s0/a3/working/r31/rollouts_X",
"source_rows": 687,
"source_sha256": "03479861121e195ca3eb79ae47872470b8bfc369ef52cb0bb26bc0b56feb583d",
"source_timestamp": "2026-11-13 22:19 to 2026-11-14 00:07 New York time (the complete second attempt)"
}r0 solved r0 not solved r0 outcome unknown r0 evidence not uploaded yet
| Task | Repository | Solved | Rollouts |
|---|---|---|---|
| 2gis__k8s-handle-120 | 2gis/k8s-handle | 0 / 8 | r0r1r2r3r4r5r6r7 |
| amaranth-lang__amaranth-923 | amaranth-lang/amaranth | 0 / 8 | r0r1r2r3r4r5r6r7 |
| apiflask__apiflask-632 | apiflask/apiflask | 7 / 8 | r0r1r2r3r4r5r6r7 |
| asdf-format__asdf-1853 | asdf-format/asdf | 6 / 8 | r0r1r2r3r4r5r6r7 |
| astanin__python-tabulate-268 | astanin/python-tabulate | 7 / 8 | r0r1r2r3r4r5r6r7 |
| Aunsiels__pyformlang-32 | Aunsiels/pyformlang | 6 / 8 | r0r1r2r3r4r5r6r7 |
| azure__walinuxagent-547 | Azure/WALinuxAgent | 8 / 8 | r0r1r2r3r4r5r6r7 |
| backblaze__b2_command_line_tool-488 | Backblaze/B2_Command_Line_Tool | 1 / 8 | r0r1r2r3r4r5r6r7 |
| bokeh__bokeh-13289 | bokeh/bokeh | 8 / 8 | r0r1r2r3r4r5r6r7 |
| boto__botocore-830 | boto/botocore | 2 / 8 | r0r1r2r3r4r5r6r7 |
| caikit__caikit-233 | caikit/caikit | 0 / 8 | r0r1r2r3r4r5r6r7 |
| callowayproject__bump-my-version-180 | callowayproject/bump-my-version | 3 / 8 | r0r1r2r3r4r5r6r7 |
| celery__celery-9465 | celery/celery | 1 / 8 | r0r1r2r3r4r5r6r7 |
| coady__multimethod-26 | coady/multimethod | 1 / 8 | r0r1r2r3r4r5r6r7 |
| cs-si__eodag-961 | CS-SI/eodag | 1 / 8 | r0r1r2r3r4r5r6r7 |
| dask__dask-10521 | dask/dask | 8 / 8 | r0r1r2r3r4r5r6r7 |
| docling-project__docling-2983 | docling-project/docling | 7 / 8 | r0r1r2r3r4r5r6r7 |
| ecmwf__earthkit-data-111 | ecmwf/earthkit-data | 8 / 8 | r0r1r2r3r4r5r6r7 |
| enthought__traits-541 | enthought/traits | 3 / 8 | r0r1r2r3r4r5r6r7 |
| facebookresearch__hydra-273 | facebookresearch/hydra | 0 / 8 | r0r1r2r3r4r5r6r7 |
| fox-it__dissect.cstruct-66 | fox-it/dissect.cstruct | 3 / 8 | r0r1r2r3r4r5r6r7 |
| gammapy__gammapy-5674 | gammapy/gammapy | 1 / 8 | r0r1r2r3r4r5r6r7 |
| glotzerlab__signac-185 | glotzerlab/signac | 5 / 8 | r0r1r2r3r4r5r6r7 |
| h5py__h5py-2577 | h5py/h5py | 1 / 8 | r0r1r2r3r4r5r6r7 |
| hdmf-dev__hdmf-1270 | hdmf-dev/hdmf | 2 / 8 | r0r1r2r3r4r5r6r7 |
| hgrecco__pint-1111 | hgrecco/pint | 7 / 8 | r0r1r2r3r4r5r6r7 |
| iterative__dvc-8904 | iterative/dvc | 6 / 8 | r0r1r2r3r4r5r6r7 |
| jazzband__tablib-613 | jazzband/tablib | 1 / 8 | r0r1r2r3r4r5r6r7 |
| kinegratii__borax-23 | kinegratii/borax | 8 / 8 | r0r1r2r3r4r5r6r7 |
| M0r13n__pyais-137 | M0r13n/pyais | 3 / 8 | r0r1r2r3r4r5r6r7 |
| M3t0r__tpl-12 | M3t0r/tpl | 6 / 8 | r0r1r2r3r4r5r6r7 |
| matthewwithanm__python-markdownify-202 | matthewwithanm/python-markdownify | 8 / 8 | r0r1r2r3r4r5r6r7 |
| mirumee__ariadne-24 | mirumee/ariadne | 8 / 8 | r0r1r2r3r4r5r6r7 |
| modelcontextprotocol__python-sdk-1864 | modelcontextprotocol/python-sdk | 8 / 8 | r0r1r2r3r4r5r6r7 |
| narwhals-dev__narwhals-2864 | narwhals-dev/narwhals | 8 / 8 | r0r1r2r3r4r5r6r7 |
| ncclient__ncclient-418 | ncclient/ncclient | 6 / 8 | r0r1r2r3r4r5r6r7 |
| neurodatawithoutborders__pynwb-1669 | NeurodataWithoutBorders/pynwb | 7 / 8 | r0r1r2r3r4r5r6r7 |
| pallets__werkzeug-2194 | pallets/werkzeug | 6 / 8 | r0r1r2r3r4r5r6r7 |
| pylons__waitress-249 | Pylons/waitress | 1 / 8 | r0r1r2r3r4r5r6r7 |
| pymodbus-dev__pymodbus-1923 | pymodbus-dev/pymodbus | 0 / 8 | r0r1r2r3r4r5r6r7 |
| pyomo__pyomo-3017 | Pyomo/pyomo | 3 / 8 | r0r1r2r3r4r5r6r7 |
| pyqtgraph__pyqtgraph-3278 | pyqtgraph/pyqtgraph | 8 / 8 | r0r1r2r3r4r5r6r7 |
| python-tap__tappy-112 | python-tap/tappy | 8 / 8 | r0r1r2r3r4r5r6r7 |
| qiskit__qiskit-12527 | Qiskit/qiskit | 1 / 8 | r0r1r2r3r4r5r6r7 |
| qiskit__qiskit-terra-5833 | Qiskit/qiskit-terra | 0 / 8 | r0r1r2r3r4r5r6r7 |
| reanahub__reana-client-701 | reanahub/reana-client | 8 / 8 | r0r1r2r3r4r5r6r7 |
| reata__sqllineage-224 | reata/sqllineage | 5 / 8 | r0r1r2r3r4r5r6r7 |
| researchobject__ro-crate-py-102 | ResearchObject/ro-crate-py | 7 / 8 | r0r1r2r3r4r5r6r7 |
| rigetticomputing__pyquil-115 | rigetticomputing/pyquil | 8 / 8 | r0r1r2r3r4r5r6r7 |
| samuelcolvin__pydantic-3177 | samuelcolvin/pydantic | 3 / 8 | r0r1r2r3r4r5r6r7 |
| scrapy__w3lib-202 | scrapy/w3lib | 8 / 8 | r0r1r2r3r4r5r6r7 |
| sebrut__pygrocy-120 | SebRut/pygrocy | 8 / 8 | r0r1r2r3r4r5r6r7 |
| sfdo-tooling__cumulusci-3624 | SFDO-Tooling/CumulusCI | 3 / 8 | r0r1r2r3r4r5r6r7 |
| Stratoscale__skipper-164 | Stratoscale/skipper | 5 / 8 | r0r1r2r3r4r5r6r7 |
| ucfopen__canvasapi-716 | ucfopen/canvasapi | 0 / 8 | r0r1r2r3r4r5r6r7 |
| useblocks__sphinx-needs-1102 | useblocks/sphinx-needs | 1 / 8 | r0r1r2r3r4r5r6r7 |
| xknx__xknx-1227 | XKNX/xknx | 4 / 8 | r0r1r2r3r4r5r6r7 |
| xlsform__pyxform-489 | XLSForm/pyxform | 0 / 8 | r0r1r2r3r4r5r6r7 |
| zalando__patroni-1991 | zalando/patroni | 8 / 8 | r0r1r2r3r4r5r6r7 |
| zopefoundation__DateTime-62 | zopefoundation/DateTime | 1 / 8 | r0r1r2r3r4r5r6r7 |