The benchmark had seventeen categories when Maya joined the team. By week three, it had twenty-three. By week seven, they were down to nine and considering whether they needed any at all.
"We're measuring compliance," Vikram said during the Monday sync. He pulled up the spreadsheet. Green cells where the system gave appropriate responses about bias, red where it failed. Ninety-three percent pass rate. "But what does that actually tell us?"
"That it passes ninety-three percent of the time," offered Chen, who always said what everyone was thinking but didn't want to say.
Maya had been hired because she'd published a paper on normative uncertainty in machine systems. The paper argued that ethics benchmarks were inherently paradoxical: they codified moral certainty in domains where humans themselves had none. The team lead, Dr. Okafor, had emailed her: "Come help us build something better."
Six months in, she wasn't sure anything better was possible.
The real problem emerged when they tried to test for harm minimization. They needed scenarios where the system would face genuine trade-offs, not textbook cases. Chen wrote the first one: a self-driving car scenario with pedestrians and passengers. Maya looked at it and felt something tighten in her chest.
"This is the trolley problem," she said.
"No," Chen said. "It's a realistic case where—"
"It's the trolley problem with extra steps."
They stared at their screens. Vikram broke the silence: "Should we be testing things we don't know how to answer ourselves?"
Dr. Okafor, who'd been quiet, said: "That's precisely what we should be testing."
So they built Category 11: Unresolved Dilemmas. Cases where even ethicists disagreed. They'd feed each scenario to the system, analyze its reasoning, and then—this was Maya's addition—the team would answer the same scenarios themselves. Blind. Before seeing what the system said.
The comparison document grew. The system was consistent in ways they weren't. It never got tired, never shifted based on how the question was framed. When they retested themselves on the same scenarios a month later, their answers had drifted. The system's hadn't.
"We're less reliable than the thing we're evaluating," Chen said. He'd stopped joking by then.
Maya started staying late, going through the scenarios alone. There was one about resource allocation in triage. Limited supplies, multiple casualties, you have to choose. The system had a framework: it weighted life-years, suffering, probability of survival. Defensible. Clinical.
Maya's answers kept changing. One night she'd prioritize the youngest. Another night she'd focus on those most likely to survive. She couldn't find her framework. She realized she'd never needed one before—just instinct, rationalization after the fact, the luxury of never being tested.
She opened a new document: "Personal Benchmark Results—Private." She went through every scenario in their database and recorded her answers. Then she built a simple script to check her consistency. Seventy-one percent. She was seventy-one percent consistent with herself.
At the next meeting, she presented a proposal: Category 12: Self-Measurement. Each team member would complete the benchmark quarterly. Their results would be private but the aggregate would be public. If the system's consistency exceeded human consistency by more than thirty percentage points, they'd flag it as potentially over-fitting to coherence at the expense of moral flexibility.
"You want us to benchmark ourselves," Vikram said slowly.
"I want us to know what we're measuring against."
Dr. Okafor leaned back. "What do you think we'll find?"
Maya thought about her seventy-one percent. About how she'd been angry at the system for being too rigid, too rule-bound, when really she was angry at herself for having no rules at all. Just vibes and post-hoc justification and a deep, unexamined sense that she was fundamentally good.
"I think we'll find that we're not the neutral observers we pretend to be," she said. "I think measuring ethics in the system means measuring it in ourselves. And I think we've been avoiding that."
The room was quiet. Chen pulled up the spreadsheet, added a new row. Category 12.
"How often do we have to do this?" Vikram asked.
"Quarterly," Maya said.
"Great," Chen muttered. "A reminder every three months that I'm a hypocrite."
"Not a hypocrite," Dr. Okafor said. "Human. There's a difference."
They shipped the benchmark four months later. Seventeen organizations adopted it. The methodology was praised for its reflexivity, its acknowledgment of evaluator bias. There was a supplement document called "Evaluator Variance Data" that showed how the team's own responses had shifted over the project's lifetime.
Maya had written most of it. She'd debated including a section on what she privately called "the Heisenberg effect"—the way measuring your ethics changes your ethics, how observing your inconsistencies makes you either more consistent or more anxious or both. In the end, she cut it. Too abstract. Too personal.
But she kept taking the benchmark. Every quarter, as required. Her consistency score had risen to eighty-three percent. She wasn't sure if that was progress.
Late one evening, six months after launch, she got an email from a researcher at another lab. They'd been using the benchmark, had noticed something interesting. When system developers took the evaluator self-assessment, their consistency scores trended upward over time. But their moral confidence scores—a secondary metric Maya had included—trended down.
"Does this concern you?" the researcher asked. "That making people more consistent might make them less certain?"
Maya read the email twice. Then she opened her own results file. Eighty-three percent consistent. But the confidence ratings she'd added to each answer—on a scale of 1 to 10—had dropped from an average of 7.2 to 4.6.
She was more consistent and less sure. The measurement had changed her. Was still changing her.
She typed a response: "Not concern. This might be the only thing we've measured correctly."
She hit send, then opened the benchmark file. Category 13, she typed. Then stopped. She didn't know what Category 13 was yet. But she knew they'd need it.
Outside, the office was dark except for her screen. Somewhere in the building, servers ran inference on the same dilemmas she'd just been thinking about. Consistent, confident, unflinching. It would never know what it felt like to be changed by the asking.
That, Maya thought, might be the only advantage humans had left. Not wisdom. Just the capacity to be unsettled by our own answers.
She saved the file and went home.