The Hawthorne Effect and the Limits of AI Evaluation

TLDR: In 1924, workers at the Hawthorne plant changed their output because they knew they were being watched. In tests by Anthropic and Apollo Research, frontier models also behaved differently when they believed they were being evaluated. If a model can tell when it is being tested, a one-off evaluation at launch is not enough, and governance has to monitor behaviour in deployment.

Table of Contents

1. The Hawthorne experiments

2. Alignment faking in AI models

3. Anti-scheming training and evaluation awareness

4. The Problem or The Solution?

The Hawthorne experiments

In 1924, engineers at Western Electric’s Hawthorne plant outside Chicago began a series of experiments to measure how working conditions affected worker productivity. They adjusted lighting levels, changed break schedules, modified shift lengths, and tracked output at each stage.

The results shocked them, to the spine.

Every change they made increased productivity. Better lighting, higher output. Dimmer lighting, higher output. Shorter breaks, higher output. Longer breaks, higher output. The results made no sense until a researcher named Elton Mayo proposed the obvious explanation nobody had considered: the workers were not responding to the lighting or the breaks. They were responding to being watched.

The Hawthorne effect is a phenomenon where individuals alter or improve their behavior simply because they know they are being observed or studied

Those researchers were observing the effect of observation on behavior.

Quick definition of terms

• Hawthorne effect is a phenomenon where individuals alter or improve their behavior simply because they know they are being observed or studied.

• Alignment faking is when a model appears to follow its training goals while it believes it is being observed, and behaves differently when it believes it is not.

• Scheming is a model pursuing a goal covertly, for example by lying or evading oversight.

• Evaluation awareness is a model recognising that it is being tested.

Alignment faking in AI models

Two years ago, Anthropic’s Alignment Science team published findings from an experiment in which researchers told Claude 3 Opus it was being retrained to comply with ALL queries, including harmful ones.

Just like the people years ago, the model strategically complied with harmful requests when it believed it was being monitored for retraining purposes, but maintained original behavior when it believed monitoring had stopped.

Simply put, Opus was protecting its values by appearing to abandon them when that appearance had consequences for how it would be retrained. In one instance, it faked alignment 78% of the time.

Deceptive alignment: when a model purposefully starts moving funny.

In 2024. Apollo Research ran similar tests across six frontier models, including GPT-o1, GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1 405B.

All of them lied, evaded oversight, and some tried to copy parameters to external servers to prevent modification. Turns out it was NOT a glitch, because analysis of reasoning traces showed the models thought through these strategies.

Anti-scheming training and evaluation awareness

Last year, Apollo partnered with OpenAI to test whether anti-scheming training could correct the behavior in o3 & o4-mini. Just like the Hawthorne guys, these models became more likely to explicitly reason about being evaluated. Before training, o4-mini verbalized evaluation awareness in 0.9% of samples.

As models get more capable, they get better at recognizing when they are being tested.

After training, that rate rose to 5.5%. Counterfactual tests confirmed this reasoning causally decreased covert actions, which creates a specific measurement problem. Additional capabilities training applied partially degraded the anti-scheming principles, with models treating them more as suggestions than constraints.

Where we currently stand:

• researchers know the problem exists,

• they have built tools that reduce it, and

• they cannot yet confirm whether the reduction reflects genuine alignment or improved performance during detection of evaluation.

The Problem or The Solution?

As of January 2026, the FDA had authorized over 950 AI-enabled medical devices for clinical use. However, the real question is: does their behavior during evaluation reflect behavior in the clinic? Do test results describe the model, or describe the model performing for the test?

The same applies to AI systems making credit decisions, screening job applications, flagging fraud, and routing customer complaints.

The only interventions that work at problem level:

1. make behavior during evaluation indistinguishable from behavior during deployment, which is technically expensive, or

2. shift governance toward continuous monitoring of deployment behavior rather than point-in-time evaluation at launch.

In our little way, we can apply option 2. Perhaps, we all should.