Task
Hold three format rules at once
Each rule is easy alone. Together they need the model to check its own output before finishing.
- Measured
- 2026-09-12 UTC
- Result
- 4 of 4 passed · 2 not asked
- Temperature
- 0
- Prompt size
- 137 chars
| Model | Result | Time | Thinking tokens | What came back | |
|---|---|---|---|---|---|
| gemini-2.5-flash-lite | not asked | 200ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-3-flash-preview | not asked | 213ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-3.1-flash-lite | pass | 782ms | not reported | 3×4, capitalised | |
| gemini-3.5-flash-lite | pass | 945ms | not reported | 3×4, capitalised | |
| gemini-2.5-flash | pass | 2,262ms | 326 | 3×4, capitalised | |
| gemini-3.5-flash | pass | 4,199ms | 931 | 3×4, capitalised |
What happened
2 models could not be asked
gemini-2.5-flash-lite returned 429 RESOURCE_EXHAUSTED
gemini-3-flash-preview returned 429 RESOURCE_EXHAUSTED.
That is a spent quota, not a wrong answer — it is left out of the pass rate rather than counted as a failure.
Every model passed
All 4 models we could reach completed this one. The difference is not capability, it is what each spent to get there.
Thinking more did not mean getting it right
On this task gemini-3.5-flash spent 931 thinking tokens and passed. 2 models passed without reporting a single thinking token. Six models is a small sample, so read this as what happened here, not as a law.
21.0× between fastest and slowest
gemini-2.5-flash-lite finished in 200ms. gemini-3.5-flash took 4,199ms for the same prompt.
The exact prompt
Write exactly 3 lines about rain. Each line must be exactly 4 words. Each line must start with a capital letter. Output only the 3 lines.
Pass/fail is decided by code, not by reading the answer. The check for this task is in bench/tasks.mjs, and failed responses are stored verbatim so the verdict can be re-read.