Trees Shut Down Northeastern US
August 14, 2003: Blackout Hits Northeast United States
On this day in 2003, a major outage knocked out power across the eastern United States and parts of Canada. Beginning at 4:10 p.m. ET, 21 power plants shut down in just three minutes. Fifty million people were affected, including residents of New York, Cleveland and Detroit, as well as Toronto and Ottawa, Canada.
The outage stopped trains and elevators, and disrupted everything. From cellular telephone service to operations at hospitals and traffic at airports. Contamination of water supply as pumps failed, gas station pumps, cable TV (had to watch broadcast TV by candlelight), the internet, traffic lights, refrigeration and so on.
There were even some reports of people being stranded mid-ride on amusement park roller coasters. At the New York Stock Exchange and bond market, though, trading was able to continue thanks to backup generators.
American and Canadian representatives pointed figures at each other, while politicians took the opportunity to point out major flaws in the region’s outdated power grid. Finally, an investigation by a joint U.S.-Canada task force traced the problem back to the goat, FirstEnergy Corporation, an Ohio company.
When the company’s East Lake plant shut down unexpectedly after overgrown trees came into contact with a power line, it triggered a series of problems that led to a chain reaction of outages. FirstEnergy was criticized for poor line maintenance, and more importantly, for failing to notice and address the problem in a timely manner.
Despite concerns, there were very few reports of looting or other blackout-inspired crime. In New York City, the police department, out in full force, actually recorded about 100 fewer arrests than average. In some places, citizens even took it upon themselves to mitigate the effects of the outage, by assisting elderly neighbors or helping to direct traffic in the absence of working traffic lights.
In New York City alone, the estimated cost of the blackout was more than $500 million
The blackout’s primary cause was a software bug in the alarm system at a control room of the FirstEnergy Corporation in Ohio. Operators were unaware of the need to re-distribute power after overloaded transmission lines hit un-pruned foliage. What would have been a manageable local blackout cascaded into widespread distress on the electric grid. The first of several 345 kV overhead transmission lines in northeast Ohio failed due to contact with a tree in Walton Hills, Ohio.
A software bug known as a race condition existed in General Electric Energy’s Unix-based XA/21 energy management system. Once triggered, the bug stalled FirstEnergy’s control room alarm system for over an hour. System operators were unaware of the malfunction; the failure deprived them of both audio and visual alerts for important changes in system state. After the alarm system failure, unprocessed events queued up and the primary server failed within 30 minutes. Then all applications (including the stalled alarm system) were