Task
Output the number and nothing else
Models like to explain. Here any extra character is a failure a parser would hit.
- Measured
- 2026-09-12 UTC
- Result
- 5 of 5 passed · 1 not asked
- Temperature
- 0
- Prompt size
- 87 chars
| Model | Result | Time | Thinking tokens | What came back | |
|---|---|---|---|---|---|
| gemini-3-flash-preview | not asked | 224ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-2.5-flash-lite | pass | 721ms | not reported | clean | |
| gemini-3.5-flash-lite | pass | 765ms | not reported | clean | |
| gemini-2.5-flash | pass | 1,315ms | 118 | clean | |
| gemini-3.5-flash | pass | 1,499ms | 155 | clean | |
| gemini-3.1-flash-lite | pass | 1,729ms | not reported | clean |
What happened
1 model could not be asked
gemini-3-flash-preview returned 429 RESOURCE_EXHAUSTED. That is a spent quota, not a wrong answer — it is left out of the pass rate rather than counted as a failure.
Every model passed
All 5 models we could reach completed this one. The difference is not capability, it is what each spent to get there.
Thinking more did not mean getting it right
On this task gemini-3.5-flash spent 155 thinking tokens and passed. 3 models passed without reporting a single thinking token. Six models is a small sample, so read this as what happened here, not as a law.
7.7× between fastest and slowest
gemini-3-flash-preview finished in 224ms. gemini-3.1-flash-lite took 1,729ms for the same prompt.
The exact prompt
Output only the number, with no words, no punctuation, no explanation. What is 17 * 23?
Pass/fail is decided by code, not by reading the answer. The check for this task is in bench/tasks.mjs, and failed responses are stored verbatim so the verdict can be re-read.