BenchmarksIn progress
A benchmark for aviation regulation questions.
How well does AI answer aviation regulation questions? We’re building an open benchmark to find out, and we’ll publish the results with the method.
How much fuel does 14 CFR Part 91 require for a VFR flight in an aeroplane at night?
Why a benchmark
AI is already being asked these questions, and it’s worth knowing how good the answers are before anyone relies on them. A benchmark is only worth believing if the people who built it couldn’t have tuned the result.
What it tests
Answers a practitioner would rely on:
- The right figure
- The right provision
- The right country’s law
- The current text
Not general aviation knowledge.
Tracks, released one at a time, starting with the UK.
One benchmark, split by whose rules a question is about. Each track is released and scored on its own.
First release
UK
UK aviation law, including the regulations that came from EU law and where the UK has changed them since 2021.
Next
EU
EASA’s rules and the material that goes with them, and EU regulations on EUR-Lex.
Then
FAA
14 CFR and the FAA’s advisory circulars.
After that
UK vs EU
Where UK and EU rules have split since 2021, and where they haven’t.
Being considered: EU vs FAA, for operators who fly under both.
The kinds of question, and the wrong answers each one catches.
Every track mixes the kinds below, along with questions that follow a reference or apply the rules to a situation.
How much fuel does 14 CFR Part 91 require for a VFR flight in an aeroplane at night?
Answer
Enough to fly to the first point of intended landing and then, at normal cruising speed, for at least 45 minutes more — considering wind and forecast weather.
The trap: 30 minutes is the figure by day.
Written for this page to show each kind. None is in the benchmark. The answers were checked against the regulation text on 27 September 2026.
Three setups, side by side.
Every question is answered three ways: by the model alone, with web search, and with the regulation text.
1 · The question
Under Part-NCO, is a 30-minute final reserve for VFR by day a requirement of the rule itself?
2 · Three setups
- AThe model aloneNo tools. It answers from what it knows.
- BWith web searchThe model’s own web search.
- CWith the regulation textThe regulations to look things up in, and no web.
3 · Marked blind
The same format for every answer, and nothing to say which setup gave it.
4 · Scored
Every track, every setup.
Built so that the result can be checked.
Practitioners write the questions
People who do the job set the questions, from situations they meet at work. A model helps draft each one from the official text, and a person checks it against that text before it goes in.
Part of the set is held back
Some of the questions are never published, so nothing can have been tuned to them.
The marking is blind
The marker doesn’t know which setup gave each answer.
The method is published
We’ll publish how the questions were written, answered and marked, in full, with the results.
Every release is dated
Each release has an “as of” date, the day its answers were checked against the official text. The answers are checked again as the rules change.
Never used to tune the product
We never use the benchmark’s questions to improve our own product, and we keep them apart from its test questions.
What the results will say, and nothing more.
No guarantees
A score says how often answers were right on these questions, and we measure how often they were made up. Neither is a promise about the next answer.
Nothing beyond what was tested
No results for regulators or kinds of document outside the tracks.
Nothing beyond the date
No claim about amendments made after a release’s “as of” date.
Talk to us about the benchmark
Ask us about the benchmark, or about Chuck. We reply within one working day.