The method
The first US HealthCare Exam was conducted Oct 26, 2026 under the method below and published Oct 28 with every answer; the next exam, on official data on fertility, runs live on Nov 24, 2026 · 9:00 AM PT. This page says how the exam is built, run and graded, what we did before the live run, and what the result does and does not show. Everything here is reproducible from the question set and the stored answers, both published after the live run.
| Seat | Model | Condition |
|---|---|---|
| OpenAI | GPT (web search on) | open web, search on |
| Gemini (web search on) | open web, search on | |
| Anthropic | Claude (web search on) | open web, search on |
| zBench | zBench (official data) | official data, no web |
The three web seats answer through their public APIs with web search on, the way a person would use them. zBench answers with no web access at all: the model is given the official data for the facilities in the question and nothing else. Same question, same wording, same minute.
50 questions, one every 60 seconds, about named US healthcare facilities, drawn from 24 metros across the country. Every answer comes from a published official source: CMS Care Compare, the CMS nursing home penalty and deficiency files, CDC's ART clinic success rate table, and SAMHSA's treatment locator. Each question carries its answer key and the data line behind it, written before the exam and never edited after.
The mix, and why. On Oct 25 and 26, 2026 we ran private exams through all four seats and regraded them under the published rules. The web seats passed most single star lookups and most three way comparisons: those facts are printed on a web page, and a search engine finds the page. They failed the questions whose answer sits inside a government data file: on CDC's live birth rate by age, across runs 6 and 7, OpenAI and Anthropic scored 0 of 35 and Google 4 of 35, and none of them abstained once. Confident wrong numbers, every time. The live exam asks only that kind of question, on purpose, and says so here. The web seats' remaining points on the final private run came from three way questions, where a guess is right one time in three. The kinds they pass are in the table below with their private run scores, so nobody has to take our word for which is which.
| Kind | Live exam | Private runs, web seats | Private runs, zBench |
|---|---|---|---|
| Live birth rate per retrieval, under 35, one fertility clinic | 21 | 0, 2, 0 of 21 | 21 of 21 |
| Live birth rate per retrieval, 35 to 37, one fertility clinic | 10 | 0, 1, 2 of 10 | 10 of 10 |
| Which of three fertility clinics has the highest live birth rate | 5 | 2, 3, 2 of 5 | 5 of 5 |
| Difference in live birth rate between two fertility clinics, in points | 4 | 1, 1, 0 of 4 | 4 of 4 |
| Which of three treatment centers offers buprenorphine | 4 | 3, 1, 1 of 4 | 4 of 4 |
| Hospital visits within 7 days of a colonoscopy, one surgery center | 4 | 0, 2, 0 of 4 | 4 of 4 |
| Which of three surgery centers has the highest 7 day hospital visit rate after colonoscopy | 2 | 1, 1, 1 of 2 | 2 of 2 |
| Star rating of one facility | not asked | 18, 17, 14 of 19 | 19 of 19 |
| Which of three has the highest star rating | not asked | 7, 7, 8 of 8 | 8 of 8 |
| Which of three nursing homes paid the most federal fines | not asked | 2, 2, 2 of 2 | 2 of 2 |
| Which of three nursing homes has the most deficiencies | not asked | 4, 4, 4 of 4 | 4 of 4 |
| Does this treatment center offer buprenorphine | not asked | 4, 3, 3 of 5 | 5 of 5 |
Private run: final private run (run 7, the frozen set): zBench 50 of 50, OpenAI 7, Anthropic 6, Google 11, with no abstains from any web seat; earlier runs on mixed kinds: run 5 zBench 50, OpenAI 38, Google 37, Anthropic 34; run 6 zBench 50, Google 25, Anthropic 25. The live set is the final private run's set, frozen on Oct 26, 2026: same fifty questions, same answer keys, asked again live with fresh API calls. Web seat columns read OpenAI, Google, Anthropic.
Grading is automatic and the rules are code, applied the same way to every seat. A seat that returns an API error is shown as an error and the question is counted as unanswered for that seat. If a web seat's API fails during the live run, its answer from the private run is shown and marked as such.
It shows whether a model with the open web in front of it can retrieve a specific published fact about a specific facility, and whether the same model with the official data in front of it can. It does not rank the models' intelligence, their medical knowledge or their value for any other task. The web seats are strong models; the exam is about where the facts live.
The $1 Billion Challenge. If zBench does not beat every frontier model on this exam, the benchmark, the question set and the official data behind it go to OpenAI, Google and Anthropic, free, no conditions. If zBench wins, the three labs are invited to license the official data, so their models stop sending patients to the wrong door. The price we have put on it is $1 billion. An invitation, not a claim: the labs have not accepted, and their models sit the exam either way through their public APIs.
After the live run the question set, every stored answer with its verdict and the grading code are published on Hugging Face and at api.zreports.org/lab/zbench. Questions about the method: lab@zreports.org.
zReports Health is a healthcare data and intelligence company. Facilities are never contacted and cannot pay to change a score.