Measurement report

Google lists 40 models. 16 of them answer.

We called every model the API advertises, one at a time, with the same one-word prompt. 9 returned 404 — they are listed, and they are not there.

Measured

2026-09-12 UTC

Listed

40 models

Key tier

Free

Prompt

Reply with exactly one word: OK

Order

Sequential

Bars scale to 2,353 ms; anything longer is clipped and marked.

Findings

Reproduce it

GEMINI_API_KEY=... node bench/probe-all.mjs

The script reads the model list from the API, calls each one in turn, and re-asks any model that returns 429 after a pause. That re-ask is not enough on a spent quota — the confirmations in this report came from calling the eight models individually, hours apart. The raw JSON carries a confirmed_solo_404 flag so you can see which verdicts were checked that way.

Caveats

One key, free tier, one location, one run per model except where noted. Latency here is a single observation and moves with all of those. 9 models remain unverified: they may be missing, or the quota may simply have been spent when we reached them. We have not guessed which.

Can they actually do the work?

We gave every model the same 9 tasks, each with a pass/fail decided by code. 33 of 37 runs passed, and 17 could not be asked (spent quota, not a wrong answer).

Per model: gemini-2.5-flash · gemini-3-flash-preview · gemini-3.1-flash-lite · gemini-3.5-flash · gemini-3.5-flash-lite

Per task: Return valid JSON that matches a schema · Answer in exactly seven words · Find one fact buried in a long document · Output the number and nothing else · Write a sentence without the letter E · Hold three format rules at once · Return an array of exactly five items · Reverse a list exactly · Four-step arithmetic, number only