🏛️ Issues and Advocacy Training
Courses › AI Issues: Technical, Legal, Social and Logistical

Lesson 9 of 9 · about 18 minutes

Checking an AI Claim: Benchmarks, Demos, Vendors and Records

You'll be able to

  • Explain what a benchmark score shows and the common ways it misleads
  • Tell a demo from evidence of real-world performance
  • Use a standard set of questions for AI vendors and government buyers
  • Know what records to request about a local agency's AI use under Idaho Code 74-102
  • Recognize 'AI washing' and the enforcement cases against it

An AI claim is a factual claim. "Our tool cuts permit processing time in half" can be checked like "our road plan saves $2 million." The trouble is that AI claims come wrapped in benchmark scores, polished demos and words like "autonomous" that make residents and officials feel they need a computer science degree to push back. You do not. You need the same questions you use on any vendor or official, plus a few specific to how AI is tested.

Reading a benchmark

A benchmark is a fixed set of test questions. A score tells you how a model did on those questions under those conditions. That is useful and narrow. The 2026 International AI Safety Report warns of an "evaluation gap": results before release often overstate usefulness in the real world, and performance is "jagged," with systems acing hard tests and flubbing simple tasks.

Common ways a score misleads:

  • Contamination. Test questions posted online can end up in training data. The model may be remembering, not reasoning.
  • Narrow task. A high score on multiple-choice medical questions is not the same as diagnosing patients.
  • Cherry-picked settings. Many attempts with the best one reported, special prompts, or tools the public version lacks.
  • Weak human baseline. "Beats humans" can mean beats crowd workers paid by the task.
  • Saturation. When every top model scores above 90 percent, small differences mean little.
  • Self-reported. The company ran the test on its own model.
The claimWhat to ask
"State of the art on [benchmark]"Which version? Compared with which models, tested when? Who ran it? Is there an independent leaderboard?
"Beats human experts"Which humans, how many, under what time limits? On which task, exactly?
"95 percent accurate"On what data? What is the error rate for each group affected? What happens in the 5 percent?
"Saves 40 percent of staff time"Measured how, over how long, against what baseline? Self-reported or timed?
"Fully autonomous"How often does a human step in? What is the intervention log?

Demos versus daily use

A demo is a performance. The vendor chose the inputs, rehearsed them, and probably ran them many times. That is normal sales practice. But a demo tells you what a tool can do on its best day, and you need to know what it does on an ordinary Tuesday with your messy files.

The METR study in lesson 6 is the cautionary tale: experienced developers believed AI made them about 20 percent faster while it measurably made them 19 percent slower. If skilled users misjudge their own results, a sales demo tells you even less. New York City's MyCity chatbot (lesson 4) was launched with fanfare and, five months later, told reporters businesses could take workers' tips.

Example: testing a transcription tool

The invented city of Pell River is considering a $14,000-a-year AI tool that transcribes council meetings. The demo was flawless. A resident who has attended every council meeting for years wants to know what the town will actually get.

  1. She picks three past meetings: a quiet one, one with crosstalk, and one with public commenters speaking Spanish.
  2. She runs 10 minutes of each through the free trial and compares against the official minutes and her own careful transcript of those 10 minutes.
  3. Counting errors: quiet meeting, 1,450 words with 22 errors, a 1.5 percent word error rate. Crosstalk meeting, 1,600 words with 128 errors, 8 percent. Spanish commenters, 900 words with 207 errors, 23 percent.
  4. Arithmetic check: 22 divided by 1,450 is 0.015; 128 divided by 1,600 is 0.08; 207 divided by 900 is 0.23.

Her finding: the tool is accurate on clean audio and much less accurate on exactly the parts of a meeting where residents speak, and the city should know that before relying on it for the record.

Questions for vendors and for governments buying AI

Checklist

  • What exactly does the tool do, and what decision does it feed? Who makes the final call?
  • What model or product is underneath, and which version?
  • What testing was done on data like ours, by whom, and can we see the results?
  • What are error rates overall and for each affected group?
  • What happens when it is wrong? How does a person find out, and how do they appeal?
  • What data goes to the vendor? Is it used for training? How long is it kept? Where is it stored?
  • Who can audit it? Does the contract allow independent testing?
  • What does it cost now, what does it cost per use, and what happens to the price at renewal?
  • What was the process before, and how will we measure whether this is better?
  • Can we stop using it and keep our data?

For federal agencies, OMB Memorandum M-25-21 already requires answers to several of these for "high-impact" AI: pre-deployment testing, a documented impact assessment, human oversight, and human review for affected people. If an agency says a tool is high-impact, ask for those documents. If it says the tool is not high-impact, ask why.

AI washing

Not everything sold as AI is. In March 2024 the SEC settled charges against two investment advisers, Delphia and Global Predictions, for claiming AI capabilities they did not have; they paid $225,000 and $175,000. The FTC has brought similar cases against consumer products. "Powered by AI" is a marketing phrase until someone shows you what the AI does.

Records requests for government AI use

Government use of AI leaves a paper trail: contracts, purchase orders, procurement scoring sheets, vendor proposals, policies, training materials, staff emails about the tool, and output logs. In Idaho, Idaho Code 74-102 gives any person the right to examine and copy public records, presumed open unless a statute says otherwise. Under 74-103, an agency must grant or deny a resident's request within 3 working days, or say in writing that it needs more time and provide the records within 10 working days. No response counts as a denial. The first 2 hours of labor and first 100 pages are free for residents. Beyond that, 74-102(10)(f) bars fees for a resident who shows all three of these: the request is likely to contribute significantly to public understanding of government, it is not mainly in the requester's own interest, and it would not happen if fees were charged because the requester cannot afford them.

Federal agencies publish AI use case inventories under M-25-21. For federal records beyond those, use the Freedom of Information Act.

Example: a records request for AI use

To the records custodian of the invented Cedar Basin School District:

Under Idaho Code 74-102, I request copies of the following records from January 1, 2025 to the present:

  1. Contracts, purchase orders, invoices and renewal notices for any software product the district uses that the vendor describes as using artificial intelligence, machine learning or automated decision-making, including tools for student monitoring, plagiarism detection, attendance risk, transportation routing or hiring.
  2. Vendor proposals and procurement evaluation documents for those products.
  3. Any district policy, guidance or training material on staff use of AI tools, including generative AI chatbots.
  4. Any validation, accuracy or bias testing reports the district received or produced for those products.
  5. Records showing the number of students flagged by any AI monitoring or risk tool, by month, and the number of flags later dismissed.

I request electronic copies. I am an Idaho resident and request a fee waiver under Idaho Code 74-102(10)(f): these records are likely to contribute significantly to public understanding of the district's operations, the request serves the public and not my private interest, and I could not pursue it if fees were charged.

Common mix-ups

  • Claim: A top benchmark score means the model is best for your use. What the evidence shows: Benchmarks are narrow, can be contaminated, and often overstate real-world performance, as the International AI Safety Report notes.
  • Claim: If staff say a tool saves time, it does. What the evidence shows: METR found a 39-point gap between how fast developers felt and how fast they were. Measure it.
  • Claim: Vendor AI tools are trade secrets, so contracts are not public. What the evidence shows: Contracts, prices and policies held by public agencies are generally public records. Agencies may withhold specific trade secret material under a cited exemption, which must be stated in a written denial.

Reporting angle

  • Start with the money. Search your city, county and school district check registers and agenda packets for vendor names you know sell AI products, and for words like "predictive," "automated," "machine learning" and "AI."
  • Pull the federal AI use case inventories for agencies with local offices and ask the local office how a listed tool is used here.
  • When a tool affects residents (benefits, policing, schools, permits), ask for the appeal process and how many people have used it.
  • Run your own small test whenever a tool is public-facing, and publish your method so readers can repeat it.

Research angle

Pictures, Numbers and Infographics

Each issue in this module leaves a measurable mark somewhere: on the power grid, in public opinion polls, in incident databases, in a federal comment docket, in business surveys. These figures put a size on the energy, copyright, safety, jobs and regulation lessons.

Women at desks working with calculators and data sheets in the NACA computer room, 1949
The "Computer Room" at the NACA High Speed Flight Station in California, summer 1949. The women at these desks were the computers: they worked out test flight data by hand with calculators. Electronic machines later took over that job title. Lesson 6 asks which tasks AI takes over next, and who does the work after. Credit: NACA (now NASA). Public domain, via Wikimedia Commons.

By the numbers

Businesses using AI, as of May 3, 2026 Information 39.7% Finance, insurance 33.9% All businesses 19.8%
Share of businesses that reported using AI in any business function, in the survey period ending May 3, 2026. The information sector runs about double the national rate. Source: U.S. Census Bureau, 2026.
More concerned than excited about AI Percent of U.S. adults 37% 2021 50% 2025
Share of U.S. adults who say they are more concerned than excited about the increased use of AI in daily life. Source: Pew Research Center, 2025.

Check yourself

1. A company says its model 'beats human experts' on a benchmark. What is the first thing to check?

2. What did the SEC charge Delphia and Global Predictions with in March 2024?

3. Under Idaho Code 74-103, how long does a state or local agency have to grant or deny a records request from an Idaho resident?

4. Which federal document requires most agencies to publish an annual inventory of their AI use cases?

5. A vendor demo shows an AI tool flawlessly transcribing a council meeting. What would best show how it performs in practice?

6. The 2026 International AI Safety Report describes an 'evaluation gap.' What is it?

Terms in this lesson

Benchmark contamination
When test questions or answers appear in a model's training data, so a high score reflects memory rather than skill.
AI washing
Overstating or inventing the role of AI in a product or service to attract customers or investors.
Demo
A prepared showing of a product, usually picked to succeed. Not a test.
AI use case inventory
A public list federal agencies must keep of the AI systems they use and what for.
Ground truth
A known-correct answer set you can score an AI system's output against.

Full glossary

Sources