TPS optimization

Replay benchmarks, not real-task outcomes. Lossy cache formats change numerics. TPS = generated tokens per server-busy second. Historical controls are not matched comparisons.

Recorded TPS over time

Report date, not execution order. Day groups are equally spaced; same-day offsets only separate points.

● Recorded TPS○ Partial / interrupted / cancelled━ Highest recorded TPS so far: 1093.9

16 plotted · 2 missing TPS. The line includes recorded partial attempts; it is not task quality. Daily maxima only, no within-day ordering. Hover/focus for details; select a point to reach its row.

18 entries · Recorded through 14-Nov-2026 · Hardware in Details
Entry	TPS	Control TPS / identity	Change	Details
Stack control · 32 lanes	476.8	Reference baseline	—	Details State Measured Report date 13-Nov-2026 Configuration 32 lanes · No speculation or stack patches Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Baseline row Window Not recorded per cell Limits Padded replay, not real task rollouts Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
Combined stack · 32 lanes	716.0	476.8 · Stack control · 32 lanes	+50.2%	Details State Measured Report date 13-Nov-2026 Configuration 32 lanes · ngram_gpu k=3; thinking budget 4096 patch; sliding-window prefix-hit fix; tuned LoRA kernels Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack ngram_gpu k=3; thinking budget 4096 patch; sliding-window prefix-hit fix; tuned LoRA kernels. Exceptions are stated above. Control Same-study recorded control Window Not recorded per cell Limits Aligned-engine C3 / lag-0 gates still required; stock decoding is not bitwise identical Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
Combined stack, async off · 48 lanes	735.5	476.8 · Stack control · 32 lanes	+54.3%	Details State Measured Report date 13-Nov-2026 Configuration 48 lanes · Same stack; async scheduling off Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Same box; lane count differs where stated Window Not recorded per cell Limits KV peak 99.9%; replay success did not transfer to full runs Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
Combined stack + fp8 KV · 32 lanes	849.6	476.8 · Stack control · 32 lanes	+78.2%	Details State Measured Report date 13-Nov-2026 Configuration 32 lanes · Same stack; fp8 KV cache Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Same box; lane count differs where stated Window Not recorded per cell Limits Lossy KV changes numerics; measurement only, not approved for training Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
Combined stack + fp8 KV, async off · 48 lanes	965.7	476.8 · Stack control · 32 lanes	+102.5%	Details State Measured Report date 13-Nov-2026 Configuration 48 lanes · Same stack; fp8 KV; async scheduling off Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Same box; lane count differs where stated Window Not recorded per cell Limits Lossy KV; KV peak 50%; not a real-rollout result Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
ngram + preserved thinking budget · 32 lanes	580.5	479.7 · Own control · budget 4096	+21.0%	Details State Measured Report date 13-Nov-2026 Configuration 32 lanes · ngram_gpu k=3; budget 4096 patch Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Same-study control Window Not recorded per cell Limits Same distribution in principle; aligned-engine checks needed Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
Prefix-hit fix + ngram · 32 lanes	701.3	Not recorded	—	Details State Measured Report date 13-Nov-2026 Configuration 32 lanes · ngram_gpu k=3; sliding-window prefix-hit fix Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control No numeric improvement claimed Window Not recorded per cell Limits Prefix hits are not bitwise recomputation on stock vLLM; C3 needed Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
Latest-match drafting + adaptive length · 32 lanes	756.4	713.7 · Combined stack before latest-match change	+6.0%	Details State Measured Report date 13-Nov-2026 Configuration 32 lanes · Combined stack; latest-match ngram and adaptive draft length Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Two matched replay pairs Window Not recorded per cell Limits Adaptive length alone within noise; replay only Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
Async staging-buffer fix · 48 lanes	760.1	Not recorded	—	Details State Measured Report date 13-Nov-2026 Configuration 48 lanes · ngram async fix; async scheduling on Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control No numeric improvement claimed Window Not recorded per cell Limits Acceptance restored to 1.90; does not remove real-rollout KV limit Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
fp8 scaling, shorter replay · 64 lanes	1075.5	Not recorded	—	Details State Measured Report date 13-Nov-2026 Configuration 64 lanes · fp8; k=3; async off; CUDA graphs to 1024; four turns Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control No numeric improvement claimed Window Not recorded per cell Limits Shorter prompts than the main bench; lossy KV Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
fp8 scaling, shorter replay · 96 lanes	1093.9	1075.5 · 64 lanes · same four-turn replay	+1.7%	Details State Measured Report date 13-Nov-2026 Configuration 96 lanes · Same four-turn setup Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Calculated from recorded TPS; different concurrency Window Not recorded per cell Limits Per-lane decode falls to 10.5; lossy KV Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
vLLM 0.32 baseline · 32 lanes	500.9	476 · Earlier v0.19.1 baseline (approx.)Historical / not matched	Not matched	Details State Measured Report date 13-Nov-2026 Configuration 32 lanes · vLLM 0.32; no speculation Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Historical engine comparison; approximate TPS Window Not recorded per cell Limits Version comparison, not an aligned-engine gate Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
vLLM 0.32 ngram · 32 lanes	599.0	500.9 · v0.32 · no speculation	+19.6%	Details State Measured Report date 13-Nov-2026 Configuration 32 lanes · vLLM 0.32; ngram k=3; bf16 KV Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Calculated from same-study recorded TPS Window Not recorded per cell Limits Replay only Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
vLLM 0.32 int4 KV · 32 lanes	560.7	599.0 · v0.32 · bf16 KV + ngram	−6.4%	Details State Measured Report date 13-Nov-2026 Configuration 32 lanes · vLLM 0.32; ngram k=3; int4_per_token_head; Triton Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Calculated from same-study recorded TPS Window Not recorded per cell Limits Lossy KV; unpacking costs outweigh smaller reads Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
FlashInfer fp8 KV · 32 lanes	811.7	617.0 · FlashInfer bf16 · same box	+31.6%	Details State Measured Report date 14-Nov-2026 Configuration 32 lanes · vLLM 0.32; FlashInfer; fp8; ngram k=3; r16; budget off Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Calculated from same-backend recorded TPS; lossy KV Window Not recorded per cell Limits Lossy KV; different engine; budget patch absent Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
FlashInfer nvfp4 KV · 32 lanes	699.6	811.7 · FlashInfer fp8 · same box	−14%	Details State Measured Report date 14-Nov-2026 Configuration 32 lanes · vLLM 0.32; text-only multimodal-prefix patch; nvfp4; ngram k=3; r16; budget off Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control Source-reported same-backend comparison; lossy KV Window Not recorded per cell Limits Lossy 4-bit KV; numerical drift unverified; more room is not proven end-to-end speed Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
64-lane combined-stack fp8 attempt · 64 lanesInterrupted	—	Not recorded	—	Details State Interrupted Report date 13-Nov-2026 Configuration 64 lanes · Combined stack; fp8 KV; async off Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control No numeric improvement claimed Window Not recorded per cell Limits Box preempted; cancelled to free GPU Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗
128-lane scaling attempts · 128 lanesFailed	—	Not recorded	—	Details State Failed Report date 13-Nov-2026 Configuration 128 lanes · Four-turn fp8 setup; k=1 / k=2 / k=3 Hardware One NVIDIA RTX PRO 6000 Blackwell (96 GB) per measurement Stack Gemma 4 31B W4A16 QAT · rank-16 LoRA · vLLM 0.19.1 · temperature 1. Exceptions are stated above. Control No numeric improvement claimed Window Not recorded per cell Limits k=1/2 shape-mismatch crash; k=3 CUDA out of memory Replay pads turns to logged lengths; four-turn scaling cells have shorter prompts. Combined stack = ngram k=3, thinking-budget patch, sliding-window prefix-hit fix and tuned LoRA kernels. Aligned-engine compatibility is not established by this measurement. Report dates do not establish exact start timestamps. Source ↗

Missing control TPS stays “Not recorded”; rounded source gains are not used to invent controls. Bundled evidence plus persisted API entries; not a live job monitor.

Public entry API · Feed / correction guide ↗