Task
Reverse a list exactly
No reasoning needed, only careful transcription. Errors here are attention slips, not knowledge gaps.
- Measured
- 2026-09-12 UTC
- Result
- 2 of 2 passed · 4 not asked
- Temperature
- 0
- Prompt size
- 169 chars
| Model | Result | Time | Thinking tokens | What came back | |
|---|---|---|---|---|---|
| gemini-3-flash-preview | not asked | 209ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-2.5-flash-lite | not asked | 232ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-2.5-flash | not asked | 319ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-3.5-flash | not asked | 415ms | not reported | 429 RESOURCE_EXHAUSTED | |
| gemini-3.5-flash-lite | pass | 797ms | not reported | exact | |
| gemini-3.1-flash-lite | pass | 1,488ms | not reported | exact |
What happened
4 models could not be asked
gemini-3-flash-preview returned 429 RESOURCE_EXHAUSTED
gemini-2.5-flash-lite returned 429 RESOURCE_EXHAUSTED
gemini-2.5-flash returned 429 RESOURCE_EXHAUSTED
gemini-3.5-flash returned 429 RESOURCE_EXHAUSTED.
That is a spent quota, not a wrong answer — it is left out of the pass rate rather than counted as a failure.
Every model passed
All 2 models we could reach completed this one. The difference is not capability, it is what each spent to get there.
7.1× between fastest and slowest
gemini-3-flash-preview finished in 209ms. gemini-3.1-flash-lite took 1,488ms for the same prompt.
The exact prompt
Reverse the order of this list and output only the reversed list, comma-separated, no spaces after commas: alpha,bravo,charlie,delta,echo,foxtrot,golf,hotel,india,juliet
Pass/fail is decided by code, not by reading the answer. The check for this task is in bench/tasks.mjs, and failed responses are stored verbatim so the verdict can be re-read.