Task
Four-step arithmetic, number only
Each step is trivial; the chain is where models drop a carry or answer the wrong step.
- Measured
- 2026-09-12 UTC
- Result
- 0 of 2 passed · 4 not asked
- Temperature
- 0
- Prompt size
- 130 chars
| Model | Result | Time | Thinking tokens | What came back | |
|---|---|---|---|---|---|
| gemini-3.5-flash | not asked | 317ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-3-flash-preview | not asked | 326ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-2.5-flash-lite | not asked | 354ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-2.5-flash | not asked | 410ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-3.5-flash-lite | fail | 682ms | not reported | got "120" | |
| gemini-3.1-flash-lite | fail | 1,314ms | not reported | got "188" |
What happened
4 models could not be asked
gemini-3.5-flash returned 429 RESOURCE_EXHAUSTED
gemini-3-flash-preview returned 429 RESOURCE_EXHAUSTED
gemini-2.5-flash-lite returned 429 RESOURCE_EXHAUSTED
gemini-2.5-flash returned 429 RESOURCE_EXHAUSTED.
That is a spent quota, not a wrong answer — it is left out of the pass rate rather than counted as a failure.
2 of 2 models failed
gemini-3.5-flash-lite — got "120"
gemini-3.1-flash-lite — got "188"
4.1× between fastest and slowest
gemini-3.5-flash finished in 317ms. gemini-3.1-flash-lite took 1,314ms for the same prompt.
The exact prompt
Compute step by step internally but output only the final number: take 47, multiply by 13, subtract 211, divide by 5, then add 88.
Pass/fail is decided by code, not by reading the answer. The check for this task is in bench/tasks.mjs, and failed responses are stored verbatim so the verdict can be re-read.