Case Studies

The Evidence

Documented failures, analyzed for structural causes.

Each of these cases is well-documented. What we provide is not new reporting but structural analysis: an examination of why the failure was predictable given the nature of the work being automated. In each case, the pattern is the same: an attempt to automate work that depended on knowledge that had never been documented.

Phoenix Payroll

$2.2B+
Nine years of operation, still not functioning correctly

The situation

The Canadian government attempted to consolidate payroll for 300,000+ federal employees into a single automated system.

The structural problem

Federal payroll involves over 80,000 distinct pay rules, most of which existed only in the knowledge of compensation specialists. The system was expected to automate decisions that had never been documented. When those specialists left or were not consulted, their knowledge went with them. The system has no way to execute rules that were never specified.

Knight Capital

$440M
45 minutes

The situation

A trading firm deployed a software update that triggered catastrophic automated trades.

The structural problem

The deployment process relied on tacit knowledge of which servers required updates. When one server was missed, the system behaved in ways that no one had specified handling rules for. The knowledge of the correct deployment procedure existed in one engineer's head. When the process deviated from what he knew, there were no documented rules for the system to follow.

Hertz/Accenture

$32M
Multiple years of failed delivery

The situation

Hertz contracted with Accenture to build a new customer-facing website.

The structural problem

Requirements were never fully specified because they existed in stakeholder preferences rather than documentation. Each review surfaced new requirements because the rules had never been made explicit. The project could not succeed because the automation was expected to implement decisions that stakeholders had not yet made.

CrowdStrike

8.5M computers
78 minutes to global propagation

The situation

A security software update caused widespread system failures across multiple industries worldwide.

The structural problem

The update validation process made implicit assumptions about production system states. When those assumptions did not hold, there were no explicit rules for handling the situation. The validation tested whether the update worked under expected conditions but not whether those conditions would actually obtain.

The Pattern

What They Share

In each case, the failure point was the same: an attempt to automate work that depended on knowledge that had never been documented. The technology was not the problem. The structure of the work was the problem. This structural characteristic is identifiable before implementation begins, which is what structural assessment does.

Assessment Before Investment

Determine whether your process has the structural characteristics that lead to these outcomes.

"Every one of these failures was predictable before investment was committed."
The Eidos Principle