Operational memory

The history was there. It wasn't on the record.

Public inquiries into major failures keep finding the same thing. What people knew about the equipment never reached the people deciding what to do with it, not because nobody wrote it down, but because of where it was written and what was left out.

Published 29 October 20266 min read8 sources

Written for maintenance and reliability engineers · plant and operations managers · safety managers

A research report written for the UK's Health and Safety Executive on ageing plant opens with a line that reframes the whole subject:

“ageing is not about how old the equipment is, but is about what is known about its condition”

Plant ageing, research report RR509 prepared for the Health and Safety Executive, 2006[1]

Equipment whose history has been lost is, in that report's terms, already further along in ageing than equipment of the same age whose history is known. It lists how history gets lost: second-hand equipment, process changes, 'changes in personnel, record keeping and other human factors'. And it warns that 'a lifetime's knowledge that has been built up by one person can be easily lost upon retirement'.^⁠[1]

The inquiries below are about far more than records. Each involved design, procedure, supervision or regulation, and we say so for each. But in every one, investigators also found that something already known about the equipment had not reached the people who needed it.

Four inquiries, the same gap

Buncefield: fourteen sticks, one unaware manager

Oil storage depot, England · 2005 · investigation report 2011

The record: A tank's level gauge stuck and the tank overfilled. The vapour cloud ignited, and the fire that followed was, in the investigators' words, the largest seen in peacetime UK. The gauge had stuck 14 times in the three months since the tank came back from maintenance. That 'was not recorded on the defects log and the Operations Manager was unaware of the frequency of failure'. The shift system 'led to short-term apparent fixing of problems with no proper overview of what was going wrong and why'.^⁠[2]
Our reading: Each stick was dealt with. None of them was counted. The independent high-level switch, which should have stopped the overfill regardless, was also inoperable; protection was the engineering failure. The pattern in the gauge was the records failure.

Texas City: a work-order system that recorded the cost, not the cause

Oil refinery, United States · 2005 · investigation report 2007

The record: Fifteen people died when a tower overfilled during start-up. The US Chemical Safety Board found the maintenance work order system 'was primarily an accounting program that lacked crucial information about work order history, the cause of failures, and whether equipment was successfully repaired'. There was no work history at all for the tower's level transmitter for the five years before the incident.^⁠[3]
Our reading: A system can hold every job and still not say what failed, what the technician saw, or whether the fix held. The Board also found the night shift's start-up had been handed over in one ambiguous log line, and that the site had no shift turnover requirement.

Piper Alpha: the valve that was off for overhaul

Offshore platform, North Sea · 1988 · public inquiry 1990

The record: The night shift set about restarting a pump without knowing its pressure safety valve had been removed for maintenance. The inquiry traced that to 'failures in the communication of information at shift handover' and in the permit-to-work system. There were no written procedures for handovers, and 'maintenance work was not always set out in the logs which were kept by operators'. Where one permit could affect another, 'reliance was placed on the memory' of the person issuing them.^⁠[4]
Our reading: The fact that mattered existed on a permit, filed elsewhere, and in the memory of the people who had done the work. It was not attached to the pump. The inquiry's deeper findings were about a permit system routinely disregarded and lessons from an earlier death that never changed practice.

Havelock North: memory that faded

Drinking-water supply, New Zealand · 2016 · Government inquiry 2017

The record: Thousands fell ill when bore water was contaminated. The same bores had been contaminated in 1998, and a report then had pointed to a leaking cable gland at the bore head. The inquiry found 'it appears nothing was learned from the July 1998 outbreak' and that 'memory of the earlier outbreak simply faded'.^⁠[5]
Our reading: The 1998 findings were a piece of the bores' own history. Kept as an open item against those bore heads, they would have outlived the people who knew them. The inquiry's centre was source protection and treatment, which no record replaces.

Where the history goes missing

Across these inquiries, and in the UK regulator's own guidance, history is lost in four predictable places.

At the shift change. The HSE notes that many accidents have followed a failure of communication at shift handover, 'the majority of these involved planned maintenance work'.^⁠[6] Its review of the published cases found logs 'were not designed to capture key information reliably and unambiguously'.^⁠[7]
In the fix that seemed to work. Buncefield's gauge was freed and the defect log was skipped, because the symptom went away. At Texas City, maintenance staff could close a job order even if the work had not been completed.^⁠[3]
In a system built for something else. An accounting system records that money was spent on a machine. It does not record the cause of the failure, what the technician saw, or whether the repair held.
In a person. The engineer who knew why a setting was changed, the operator who knew the gauge stuck. When they move on, the history goes with them unless it was written where the next person will look.

What the investigators asked for

The recommendations are notably practical. The Chemical Safety Board asked the refinery to:

“Establish an equipment database that captures the history of testing, inspections, repair, and successful work order completion. Analyze repair trends and adjust maintenance and testing intervals to prevent breakdowns.”

US Chemical Safety Board, recommendation 2005-4-I-TX-R15[3]

The Buncefield authority concluded that 'management should have in place systems to monitor the reliability of safety critical equipment'.^⁠[2] The HSE's ageing-plant guidance asks for a maintenance log 'specific to each item of equipment', recorded so as 'to differentiate between 20 new seals on one pump, and 20 new seals over several pumps', and adds that 'a brief statement of actions is normally all that is needed'.^⁠[1]

What this means for a maintenance record

Attach history to the equipment, not to a filing location. A permit in a rack, a log book in a control room and a spreadsheet on a drive are three places the next person will not look together.
Record what was found and whether it worked, not only that a job closed. Research on Australian mobile-equipment work orders found the coded fields often wrong or missing, and the technicians' free text was what made the history usable.^⁠[8]
Count the repeat. The second occurrence of a fault on the same equipment is information. The fourteenth is a pattern someone should have been shown.
Carry open items across the shift. An isolation, a removed valve or a temporary override should stay visible on the equipment until someone closes it, not be re-copied from one log page to the next.

How Mainte keeps what was learned on the machine is set out in What the last person knew. How that history turns a run of repairs into a decision is in The fault that keeps coming back.

Sources

[1]Plant ageing: management of equipment containing hazardous fluids or pressure (RR509) · Health and Safety Executive (UK); prepared by TWI, ABB Engineering Services, SCS and Allianz Cornhill Engineering · 2006 · Research report commissioned by the regulatorQuoted from the summary and sections 2.4.2, 2.5.2 and 3.1.1. The HSE has since moved its research pages; this is the Internet Archive copy. Its contents are the authors' and not necessarily HSE policy.
[2]Buncefield: Why did it happen? The underlying causes of the explosion and fire at the Buncefield oil storage depot · COMAH Competent Authority (HSE, Environment Agency, SEPA) · February 2011 · Major incident investigation reportQuoted at paragraphs 26, 54, 56 and 58. Copy hosted by IChemE; the official HSE copy is in the UK National Archives.
[3]Investigation Report: Refinery Explosion and Fire, BP Texas City (Report No. 2005-04-I-TX) · US Chemical Safety and Hazard Investigation Board · March 2007 · Federal incident investigation reportQuoted at sections 1.5, 7.2.3, 7.2.4 and 10.1.1, and recommendation R15 (p. 215).
[4]The Public Inquiry into the Piper Alpha Disaster, Volume 1 (Cm 1310) · The Hon. Lord Cullen, HMSO; republished by the Health and Safety Executive · November 1990 · Public inquiry reportQuoted at paragraphs 1.5, 11.4 and 11.13.
[5]Report of the Havelock North Drinking Water Inquiry: Stage 1, part 1 overview · Government Inquiry into Havelock North Drinking Water, Department of Internal Affairs (NZ) · May 2017 · Government inquiry reportQuoted at paragraphs 13 and 14.
[6]Human factors: shift handover · Health and Safety Executive (UK) · Current guidance · Regulator guidance
[7]Effective shift handover: a literature review (OTO 96 003) · Health and Safety Executive, Offshore Safety Division (UK) · June 1996 · Regulator-published literature reviewQuoted from section 4.7. Internet Archive copy of the HSE original.
[8]Cleaning historical maintenance work order data for reliability analysis · M. Hodkiewicz and M. Ho, University of Western Australia; Journal of Quality in Maintenance Engineering 22(2) · May 2016 · Peer-reviewed journal articleCited from the published abstract; data from heavy mobile equipment across several organisations.