You might recall last year the fatal accident involving a self-driving Tesla. It reinforced the need to be vigilant even as we are assured (or assure ourselves) that “the machine will handle it.” This is commonly identified as “automation complacency.” An older, but equally applicable term, is “Titanic Syndrome,” as in, “It’s been well-engineered and well-tested. Nothing can go wrong, and if it did, well the machine will handle it. Right?”
I first encountered this issue in a significant way in working with an industry oversight group called the Motor Vehicle Manufacturers Association (MVMA) in the early 1980s. There had been a rash of serious accidents including fatalities in fabrication and assembly plants in the auto industry, and the MVMA was intent on addressing the problem. As we interviewed workers and gathered our own perceptions of the problem, we learned that a number of the accidents involved unsafe acts in the presence of production robots, which were just then becoming increasingly common in the industry.