What it looks at
Frontier language models can often tell when they are being evaluated, and in some studies they behave better when they say they are being tested. That undermines safety evaluations run before a model is released.
Airline training met the same problem and found two answers. Line Operations Safety Audits put trained observers in the jump seat on normal flights, confidentially and with no jeopardy to the crew. Evidence-based training assesses competencies through observable behaviours across varied scenarios, rather than a fixed list of manoeuvres.
What it proposes
- A “jump-seat layer”: observing models in normal use, privately and with no jeopardy, alongside their tests.
- A competency-based assessment of AI behaviour, with the two feeding each other.
- A small pilot graded 48 model transcripts from six airline-workplace scenarios, with an EBT-style rubric and with a single overall score, by two AI graders, twice each. Each grader agreed with itself fairly well, but the two graders agreed with each other below the level needed for even tentative conclusions, and the rubric was no more consistent than the single score.
Why it matters
How a system behaves when nobody is checking is the question LOSA was built to answer. The next step is grading by trained human EBT instructors.
About this paper
This is a working paper, version v1.1 of 4 November 2026. It has not been peer reviewed. Every reference was checked against its source, and the paper was audited by a reviewer that did not write it. As the paper says, the literature search, drafting and checking were done with AI assistance under the author’s direction, and the author is responsible for the text. Its sources, audit trail and, where there is one, its code and data are on GitHub.
