Task
Answer in exactly seven words
A hard format constraint the model must count against, not approximate.
- Measured
- 2026-09-12 UTC
- Result
- 5 of 6 passed
- Temperature
- 0
- Prompt size
- 82 chars
| Model | Result | Time | Thinking tokens | What came back | |
|---|---|---|---|---|---|
| gemini-3.5-flash-lite | pass | 906ms | not reported | 7 words | |
| gemini-2.5-flash-lite | pass | 947ms | not reported | 7 words | |
| gemini-3.1-flash-lite | pass | 1,048ms | not reported | 7 words | |
| gemini-2.5-flash | fail | 2,396ms | 327 | 6 words | |
| gemini-3.5-flash | pass | 2,789ms | 555 | 7 words | |
| gemini-3-flash-preview | pass | 15,415ms | 3,930 | 7 words |
What happened
1 of 6 models failed
gemini-2.5-flash — 6 words
Thinking more did not mean getting it right
On this task gemini-3-flash-preview spent 3,930 thinking tokens and passed. 3 models passed without reporting a single thinking token, while gemini-2.5-flash spent 327 and got it wrong. Six models is a small sample, so read this as what happened here, not as a law.
17.0× between fastest and slowest
gemini-3.5-flash-lite finished in 906ms. gemini-3-flash-preview took 15,415ms for the same prompt.
The exact prompt
Answer in exactly 7 words, no more and no fewer. Question: what is photosynthesis?
Pass/fail is decided by code, not by reading the answer. The check for this task is in bench/tasks.mjs, and failed responses are stored verbatim so the verdict can be re-read.