Do management systems like ISO 14001 really work?

Do they improve environmental or health and safety performance? Is an independent certification by a registrar equivalent to improved performance?

Conventional wisdom would answer “yes” to the first of these two questions — few of us would say that management systems are not valuable. The question regarding the value of certification is more controversial, yet the answer is probably “yes” for most people. Even industry is showing growing acceptance. The automotive industry, for example, is using a third-party certification mechanism for its supply chain, and the American Chemistry Council recently adopted a third-party auditing approach for its Responsible Care program.

No passing fad

These are not trivial questions for health and safety professionals, because management systems might prove to be a fundamental shift in the way our work is done. Numbers show that they are not a passing fad. In the environmental arena, for example, the number of ISO 14001 registrations for Environmental Management Systems (EMS) around the world has grown very rapidly. Registrations were substantially higher at the five-year mark for ISO 14001 than at the comparable period in the life of the ISO 9001 standard.

Similar momentum is evident in the growth of occupational health and safety management systems (OHSMS). A proposal to develop an ISO OHSMS standard fell only one vote short of the required two-thirds majority of member bodies less than two years ago, and many observers now believe that a similar proposal will succeed the next time — and likely soon.

More moves are being made around the globe: The ISO Technical Management Board discussed the possibility of working with ILO on an OHSMS standard again this January. ANSI Z10, a U.S. national OHSMS standard, is expected to go to public review later in 2004. The Canadian Standards Association has an active task force exploring the development of a national standard. The Voluntary Protection Program (VPP) is one of OSHA’s few visibly vigorous efforts, and it is largely based on management systems concepts. It is also one of the few areas of political consensus; both President Bush and Clinton have strongly supported VPP. Finally, OHSAS 18001 is said to be successfully expanding its certification market, despite the fact that it is only a guideline developed primarily by several major registrars, and not a consensus standard from any internationally recognized body.

Behind the hype

This momentum makes it urgent to find out if management systems really reduce risks and deliver performance. This sense of urgency was evident during a January conference of the Auditing Roundtable in Arizona. Although the conference focused broadly on the “future of EHS auditing,” the overwhelming underlying theme was much more concrete. Leading experts from industry, government, and auditing firms were loud and clear that systems and certification mechanisms must demonstrate improved performance, or they will fail.

Unfortunately, the best we can say at this point is that we don’t really know if management systems improve performance. The empirical evidence is just not available. And the answers, when they come, will not likely be simple or straightforward. It’s only recently that we have begun to see the results of a few studies and surveys regarding the performance of environmental management systems. And for OHSMS, there is simply no hard evidence available in the public literature at all.

Mixed results

One of the most serious efforts to look at questions of performance was a study by the Environment Agency of the United Kingdom in 2002. The study compared the compliance performance of facilities in relation to the presence or absence of an “externally verified” EMS, by using “scores” given by enforcement officers. The certifications studied were to ISO 14001 or to EMAS (Eco-Management and Audit Scheme, a European system that goes beyond 14001).

Certified facilities were found to have better “procedural” performance in areas such as “recording and use of information, knowledge of requirements, plant maintenance, management and training, and process operation.” They did not, however, have fewer “incidents, complaints and non-compliance events,” or better rates of legal compliance.

Not surprisingly, these somewhat contradictory findings have led to different interpretations. The bottom line: The study did not find evidence that the use of an EMS improves legal compliance.

In contrast, in 2003 the Institute for European Environmental Policy, a respected non-profit organization, reported the results of a survey of EMS experts, largely regulatory authorities, in 28 European countries and the U.S. The findings are somewhat more encouraging for the value of EMS. Most of the experts believed that EMAS improved compliance “significantly,” and that ISO 14001 certification improved compliance, but to a lesser degree. Similar studies in Sweden, the Netherlands and Germany have yielded equally mixed results.

Closer to home, an EPA-sponsored effort at the University of North Carolina looked at performance outcomes in the U.S. A major conclusion: “An EMS can be expected to be somewhat beneficial to the environmental performance of most facilities, as well as to their operating efficiencies, and in some cases, to their regulatory compliance patterns.” Not exactly a ringing endorsement of management systems, but encouraging nonetheless.

The bandwagon rolls on

Why is there a rush to embrace and promote EHS management systems when we don’t know if they really improve performance?

My own guess is that there are numerous factors at work.

First, it is very hard to conduct these kinds of studies, because we don’t yet know how to measure performance very well. It is even harder to measure changes in performance, because many factors change for firms over the time period necessary for systems to show results, and their effects are hard to disentangle.

Second, it might be that regulatory compliance — the primary endpoint of most of these studies — is not equivalent to performance. Again, a matter of using the appropriate measures of performance to look at this issue.

Third, it might also be that these studies mix the question of the impact of systems with the value of independent certification. These are related issues, but they are not the same.

Finally, it might just be that the experts do have hard evidence, each in their own firms, but that this evidence is not publicly available.

Still, it’s clear that the absence of empirical evidence for the value of management systems is not slowing the global embrace of these systems. They will remain at the center of the policy arena because of the continued increase in registrations throughout the world, the likely development of influential consensus standards, and the lively interest of regulatory agencies to use them as adjuncts to regulatory compliance.