Averting Catastrophe: Chapter 7: A System for Averting Catastrophe

Averting Catastrophe
source ref: ebookcat.html

	Chapter 1:The Potential for Catastrophe
	Chapter 2:Toxic Chemicals
	Chapter 3:Nuclear Power
	Chapter 4:Recombinant DNA Research
	Chapter 5:Threats to the Ozone Layer
	Chapter 6:The Greenhouse Threat
	Chapter 7: A System for Averting Catastrophe
	Chapter 8: Can We Do Better?
	Notes

Chapter 7: A System for Averting Catastrophe

7
A System for Averting Catastrophe

This volume began with an apparent paradox. We are surrounded by potentially catastrophic threats from civilian technologies and yet there has been no catastrophe. To what do we owe our good fortune? In examining five types of technological risks that pose a potential for catastrophe, it appears that our good fortune was due in part to luck, in part to delayed consequences yet to be faced, and in part to jerry-rigged solutions. A not insignificant part of the explanation, however, is that regulators have coped with risky technologies in a surprisingly intelligent manner. That is not to say that the outcomes are fully satisfactory; nonetheless, each risk has been approached using one or more sensible strategies. Moreover, the individual strategies at times cohere well enough to seem almost like a system.

The Strategies Now in Use

Use of toxic substances originally proceeded by trial and error, and chemicals were regulated only after negative consequences became apparent. This type of decision process is a well-known, thoroughly analyzed strategy for coping with

122

complex problems (see the third section in this chapter). But we had assumed that long delays before obtaining feedback, coupled with severe consequences of error, would make trial and error inappropriate for managing hazardous chemicals. Contrary to our expectations, there proved to be numerous ways to obtain feedback about the effects of chemicals, as demonstrated in the case of pesticides, and regulators were able to take repeated corrective action in response to this feedback.

In the past several decades, however, the number of chemicals introduced into the environment and the number of people exposed to them has increased exponentially. The strategy of waiting to experience effects before taking action became less acceptable, and more deliberate steps were initiated. Two basic approaches evolved, both intended to prevent or reduce severe health and environmental consequences. First, new chemicals must now undergo a premanufacture notification and screening process that attempts to identify the most hazardous substances before they are marketed. Second, because the sheer number of existing chemicals prevents attention to all, priority-setting procedures identify those chemicals that most need testing and regulation.

In contrast to the case of toxic substances, regulation of nuclear power was never based on normal trial and error, even in its earliest days. The potential consequences of errors in design, construction, and operation were obviously unacceptable, yet the complexity of reactor designs made errors unavoidable. Nuclear regulators seem to have been aware of this dilemma from the early days of reactor development; their solution was, and still is, to attempt to make reactors forgiving of errors. They assumed that errors would occur and required that reactors be designed to withstand such errors. First, reactors were designed conservatively to prevent errors in design, construction, and operation from leading to a release of fission products from the core. This was achieved through wide margins for error, redundancies, and emergency systems. Second, reactors were designed to minimize the effects of accidents should they occur despite the attempts to prevent them. The main tactic to achieve this was containment. Over time, the primary emphasis in regulation has shifted to preventing core melts, and away from minimizing their effects.

123

The approach employed by the National Institutes of Health in regulating recombinant DNA research combined the strategies used for regulating nuclear power and toxic chemicals. On the one hand, an effort was made to make rDNA research forgiving of errors. Both physical and biological containment were required, so that if an organism were released during an experiment, it would be very unlikely to escape from the lab and establish itself in the environment. Having ensured protection from the consequences of error, policy makers then proceeded by trial and error. They initially established relatively stringent regulations prohibiting six classes of experiments and requiring all others to be performed under varying degrees of containment. Gradually, as experience in recombinant DNA research grew, more experiments were allowed at lower levels of containment. Eventually, all the prohibitions and most of the containment requirements were dropped.

Some critics suspect that the NIH and the scientific community have been dishonest or biased about the risks associated with rDNA research. We find their arguments unpersuasive; but even if they are correct, the strategy that evolved for dealing with the rDNA problem was well suited to the nature of the problem that faced the NIH in the 1970s. In fact, it was identical to the strategy of nuclear decision makers in establishing their first regulatory policies. Instead of being relaxed over time, however, nuclear regulations gradually have been tightened. In our view, the difference in the fates of rDNA research and nuclear power is due to differences in the natures of the two problems. The risks of rDNA research were inherently more containable and more testable than those of nuclear power.

The ozone and greenhouse cases exhibit another pattern for averting catastrophe. In contrast to nuclear power and rDNA, no one suspected at the outset that there would be any harmful effects to the atmosphere. Thus, chemicals that deplete the ozone layer were released and fossil fuels were burned for many years before the possible problems were recognized. In contrast to toxic chemicals, the more diffused and subtle nature of the atmospheric threats prevented negative feedback from serving as a warning. Instead, some scientists predicted errors on the

124

basis of scientific theories and atmosphere-climate computer simulation models. These scientists made their findings public, which led to media coverage and scrutiny by fellow scientists. These public revelations stimulated inquiry and funding of research by government and led to further scientific analysis of the threats.

Regulatory actions against atmospheric threats are even more difficult to devise than those for other risky technologies. Because ozone depletion and climatic warming are global phenomena, containment of the hazard is impossible, and only limited testing is practical. Furthermore, no one nation can do much to reduce the hazards, yet not all nations have the same incentive to act. So cooperative international action, while required, is improbable.

In the ozone case, the United States banned two of the major threats that seemed to pose the greatest risk with the fewest benefits: the SST and fluorocarbon aerosols. The implicit strategy was to take partial, readily available, and relatively low-cost steps to protect against the potential hazard. More extensive (and therefore more costly) actions were delayed until uncertainty about the likelihood and severity of the problem could be reduced through further scientific monitoring. Few other nations adopted this approach, and even the United States did not take action against other ozone depletion threats. As a result, the total quantity of ozone depleters released worldwide is now as high as it was at the height of the ozone controversy. Fortunately, it appears that other atmospheric phenomena at least partially offset this problem. The extent of the projected damage is still in dispute, but it may be somewhat less than originally expected.

No nation has yet taken action (beyond research) against the greenhouse threat. Such action would face many of the same obstacles confronted by efforts to stem ozone depletion. Moreover, the threat of climatic warming is a result largely of activities that are fundamental to a highly populated, affluent civilization. So while there are policy options available (such as reforestation or a coal ban) to counteract the greenhouse threat, these would be very expensive and politically unattractive. Mitigating the worst possible effects (for instance,

125

through crop research) is the least expensive option in the short term, and therefore the most politically feasible.

Toward a Catastrophe-Aversion System

What do the five cases studied in this volume imply for the overall goal of averting catastrophes? Political scientist Todd LaPorte has noted that regulators of many risky technologies must strive for freedom from error; this has been nearly achieved in air traffic control.^[1] He cautions that the training, design, and other requirements for error-free operation of risky technologies will be difficult to achieve. We go further than LaPorte in perceiving obstacles to error-free risk management. Except for a very few special cases where nearly complete information is available and the possible ways to err are limited, errors are unavoidable in the management of complex technologies. Air traffic control is one of the exceptions to this and not a model that can be applied generally. In the cases we have studied, freedom from error is not a realistic goal. The nature of potential errors is too uncertain, especially at the outset, to expect to prevent them all. It is this high degree of uncertainty, combined with the potential for catastrophe, that makes the cases so problematic. Errors cannot be avoided, yet errors can lead to catastrophe.

The strategies we found for coping with this dilemma were not fully developed, nor always implemented effectively. However, taken together they suggest the elements of a complete catastrophe-aversion system. The system is by no means mature, and it is never complete in any of our five cases. But the general structure can be discerned fairly readily and unambiguously.

Strategy 1
Protect against the Potential Hazard

If errors are inevitable and can lead to catastrophe, then the first priority is to protect against the worst consequences that might result from errors. We found five interrelated types of tactics for accomplishing this goal.

126

Containment of the effects of what might otherwise be a catastrophic accident was employed in the early nuclear and recombinant DNA research cases. In both cases, regulators believed that containment would make acceptable what otherwise could be serious accidents. When it can be achieved, this is the ideal solution to the dilemma. In effect, it eliminates the potential for catastrophe. Unfortunately, as we have seen, this goal usually is unattainable.^[2] In the atmospheric cases, misguided policies can result in uncontainable global climate changes; use of toxic substances is too widely dispersed to allow for containment. And if containment is feasible at all for large conventional nuclear reactors, the cost would be prohibitive.

The safest (and costliest) alternative to containment is to prevent errors entirely by prohibiting the action or technology that poses the potential for uncontainable catastrophe. A less drastic measure is to impose a selective ban on risky technologies such as the screening of toxic chemicals, early prohibitions on certain classes of rDNA experiments, and the elimination of most fluorocarbon aerosols. A still weaker variation of this strategy is to limit use of the technology to levels that are presumed to be safe or safer. One example of a way to head off the greenhouse effect would be to limit the amount of high-carbon fuels that can be burned. Another example, proposed but not implemented by EPA, would be to limit the amount of ozone-depleting chemicals that can be manufactured. The equivalent strategy applied to nuclear reactors would be to limit the size of reactors, their number, or the geographical areas in which they could be built.

Another tactic for protecting against potential hazards is to assume that errors will occur and take steps to prevent those errors from resulting in hazardous outcomes. This tactic was emphasized for reactors built after 1966, when errors still were inevitable but containment was no longer considered guaranteed. Substantially upgraded emergency core cooling systems, for example, were used in an effort to prevent reactor coolant leaks from triggering core melts. Another example of this tactic would be to attempt to offset the effects of CO₂ emissions by such measures as reforestation. This approach does not rely

127

Table 1
Strategy One: Protecting Against Potential Hazards

on the dubious hope of preventing all errors (although efforts are made to avoid them) but instead emphasizes preventing the effects of errors from producing a hazardous outcome; the point is to intervene in the sequence of events between error and severe consequence.

A final tactic is to assume that errors will occur and will trigger hazardous outcomes but to take steps that acceptably mitigate the impact. Many observers believe that this will work with the greenhouse effect because humans will adapt to a warmer and drier climate. Other examples of this tactic include remote siting of reactors (in the early nuclear era) and the proposed use of potassium iodide pills to prevent cancer of the thyroid in the event of a nuclear power plant accident. However, mitigation of effects is usually a supplemental strategy, not a primary method for averting catastrophe.

Table 1 summarizes the strategies for protecting against potential hazards used in these cases. It reveals that there are a number of points along the chain of events between error and catastrophe where regulators can intervene in order to protect against the catastrophe. At one end of the chain is the zero-risk option: prohibit the use of the risky technology. At the other end is catastrophe mitigation: the accident occurs and steps are taken to reduce its effects. Typically, prohibition is impractical and mitigation is incomplete. Some combination of the three intermediate strategies thus becomes necessary.

Strategy 2
Proceed Cautiously

We rarely know in advance just how bad or how likely a hazard might be. On what basis, then, can policy makers decide whether to make protective measures tight, lax, or somewhere in between? Some strategy is required. In protecting against a potentially catastrophic threat, the second strategy is to err on the side of caution.

The nuclear case provides several illustrations of this approach. Early on, when reliance was placed on containment, acceptable reactor designs were based on judgments of whether the containment building could withstand the maximum credible accident. It would have been possible to be less cautious and to require that containment designs withstand only the most likely accidents. Instead, a more conservative approach was taken: assume the worst and design to withstand it. Similarly, reactors were required to withstand higher than likely temperatures and pressures and were built with several levels of redundancies. Even the redundant systems such as the emergency core cooling system were designed to withstand higher than expected temperatures and pressures. Since caution is a matter of degree, some critics of nuclear power argue that decision makers should have been even more cautious.

Another important element of a cautious strategy is how the burden of proof is determined and on whom it falls. At one extreme, new technical activities could be considered dangerous until proven otherwise. Even faint suspicions of danger would be adequate reason to withhold approval, and the burden of proving safety would rest entirely on the party seeking to undertake the activity. This was the case with recombinant DNA research in the mid-1970s when elaborate and stringent precautions were taken. At the other extreme (approached earlier in this century), new activities would be considered safe until proven dangerous. The government would be required to prove danger, and only compelling evidence would be sufficient to slow or stop a technical activity. Over the past decades, the burden of proof has shifted significantly toward the proponent of an activity a more cautious approach to policy. Who should bear the burden of proof always is a matter of judgment.

129

In the case of toxic chemicals, the most striking example of this conservative approach is the Delaney Clause, which prohibits additions of any carcinogenic substance to food, even if there are compensating benefits, even if the substance is only weakly carcinogenic, and even if only trivial amounts of the substance are present. Moreover, recognizing that it is very difficult to prove that a chemical causes cancer in humans, advocates of this policy assumed that any animal carcinogen is also a human carcinogen, even though there are some that are not. They explicitly stated during congressional deliberations that when regulating food additives it is better to err on the side of caution. Because it is so extreme, however, the Delaney Clause has rarely been applied and probably is no longer realistic. As measurement capabilities have improved, virtually all foods now can be shown to contain at least trace amounts of questionable chemicals, and the continued use of saccharin and nitrites are two of several examples of possibly carcinogenic substances that continue to be added to foods in order to gain substantial benefits.

Current pesticide regulations also mandate caution, including explicit requirements for manufacturers to bear the burden of proof that a pesticide is safe enough. But this cautious approach becomes difficult to apply in practice. First, Congress requires that EPA evaluate a pesticide's risks against its economic benefits. Second, most pesticides now in use were approved before the current regulations took effect. Moreover, EPA has insufficient staff to carefully scrutinize more than a few dozen pesticide chemicals each year, so a strategy of proceeding cautiously has been adopted in principle but it has not been fully implemented in practice.

Policy on the greenhouse effect to date has not been conservative. In this case, the issue is not how extensive to make the protections but whether to take any precautions at all. Combustion of fossil fuels and production of greenhouse gases have proceeded as if there were no threat, and the burden of proof is on those who challenge these risk-producing activities. On balance, this may be appropriate considering the uncertainties about the greenhouse threat, the benefits of using fossil fuels, and the costs of corrective action. But given the conceivable

130

severity of the consequences, current policy may not be cautious enough. There is a strong temptation to discount future costs of the greenhouse threat in comparison to the near-term costs of preventive action, particularly in view of the unattractive set of alternative actions proposed to date.

The United States and a handful of other nations have proceeded more cautiously against the ozone threat. The SST and aerosol fluorocarbons were banned on the basis of scientific theories, even though there was no direct evidence of harm. Some manufacturers protested the action and called attention to the economic costs, but the majority of atmospheric experts and policy makers found the potential harm sufficiently grave to justify considerable caution. No nation pursued this policy to the fullest, however. Fluorocarbon refrigerants, degreasing solvents, and a variety of chlorocarbon and bromocarbon products continue to be used initially because they were considered more essential than aerosols, subsequently because the magnitude of the risk appeared to decline.

To reiterate, caution is a matter of degree. Even when policy makers proceed conservatively, they inevitably make controversial judgments. And there will always be dissenters, some with carefully reasoned arguments, who believe that more (or less) caution is warranted. We will consider the issue of "How cautious is cautious enough?" in chapter 8.

Strategy 3
Test the Risks

Once conservative measures for coping with the potential hazard are taken, the next step is to reduce uncertainties about the hazard's likelihood and magnitude. One way of doing this is by learning from experience (see strategy 4). An alternative approach for reducing uncertainty is to test for or simulate the hazard under controlled conditions. Unfortunately, as we saw in comparing nuclear power with rDNA research, the uncertainties associated with some hazards are more amenable to testing than others. Testability, like caution, is a matter of degree. At one extreme are the ozone and greenhouse problems; there is no way to realistically simulate these global atmospheric phenomena. At the other extreme is

131

rDNA research, where worst-case scenarios could be simulated under well-controlled laboratory conditions.

Toxic chemicals and nuclear reactor safety are cases that fall in the intermediate range. Toxicology in the 1980s bears little resemblance to toxicology of the 1940s, and the capacities of this field are even far ahead of what they were just a decade ago. Short-term screening tests for mutagenicity, analysis of chemicals' effects based on their molecular structures, and powerful new techniques for detecting minute quantities of chemicals are among the improvements that have contributed to toxicologists' ability to discern hazards. Nevertheless important limitations remain. Much of the testing is done with animals; we assume that animal reactions to toxic substances closely approximate those of humans, but we cannot be sure. In addition, it is not feasible to fully test all chemicals (at present more than sixty thousand). Only about two thousand chemicals have been tested adequately for carcinogenicity, fewer for other chronic effects such as liver damage. Even new chemicals are not being tested exhaustively, although all are evaluated to some extent.

The limitations on testing in the case of nuclear power are entirely different. To begin with, there is a matter of scale: in order to simulate a serious reactor accident, a very remote area must be used and a large reactor must melt down. If the critics of nuclear power are correct, the results of such testing could be long-lasting and widespread. However, these considerations by themselves might not be sufficient reason to reject a deliberate meltdown as a means of gaining knowledge and reducing uncertainty. The more important problem is whether we would learn enough from a single meltdown to make the risks worthwhile. Since there are many courses a meltdown could follow and only a small number of possibilities that would occur in a single test, the information gained from even several meltdowns probably would be inconclusive.

To confront these difficulties, one tactic has been to simulate aspects of serious accidents under controlled conditions. Throughout the history of reactor development, relatively small-scale tests have been performed. For example, in the early 1950s a series of experiments were run in which water in

132

small experimental reactors was deliberately allowed to boil. At the time it was feared that boiling water would make reactors unstable and difficult to control, but the experiments showed otherwise. Based on these results, the Atomic Energy Commission and the nuclear industry began to design boiling water as well as pressurized water reactors.

A more recent example of ongoing testing was a July 1985 reactor test made at Idaho Falls. A small reactor was deliberately subjected to a loss of coolant with the objective of obtaining a better understanding of the fission products that are released in serious accidents. This objective was achieved, and the test served its purpose.^[3] Many other such tests have been conducted, and they have been very useful for narrowing uncertainties about specific aspects of reactor behavior. But such specialized and limited tests cannot eliminate large uncertainties about overall nuclear risks.

Strategy 4
Learn from Experience

An alternative to testing is to learn from experience. This is accomplished by monitoring mishaps that occur despite precautions and by taking steps to prevent such mishaps from recurring. The classic trial-and-error strategy for dealing with complex problems is to: (1) establish a policy, (2) observe the effects of that policy, (3) attempt to correct for any undesired effects, (4) observe the new outcome, and (5) make corrections again. Obviously, regulators should not rely entirely on this strategy (as they did initially in toxic chemicals regulation). But once steps have been taken to protect against potential catastrophe, learning from experience via trial and error is appropriate as a supplemental strategy to reduce uncertainty.

The history of nuclear regulation is replete with examples of trial-and-error learning. Many changes in the regulations governing operator training, design of reactor control panels, operation, maintenance, and emergency procedures evolved in response to the lessons learned from the Three Mile Island accident. While TMI is an extreme case, it is by no means an exception. Regulatory changes in response to reactor incidents have been the rule in the history of nuclear regulation so

133

much so that the nuclear industry and some policy analysts have criticized regulators for overreacting to these incidents.

The same pattern of learning from experience emerges in other cases. Relatively stringent safety guidelines were established in the mid-1970s for rDNA research and then were gradually relaxed. This was partially in response to the results of testing but also partially in response to actual experience with rDNA experimentation. Regulators likewise have learned from experience in toxic substances control. For example, the discovery of badly flawed and even fraudulent toxicology testing has led government agencies to conduct routine audits of independent testing laboratories, and the Interagency Testing Committee has learned from experience to recommend individual chemical substances for testing rather than broad categories of substances.

While learning from experience plays a prominent role in the cases discussed, it nevertheless is the least developed and most poorly implemented of the catastrophe-aversion strategies. Learning from experience too often has been a purely reactive strategy regulators wait for errors to emerge, then make corrections. In a well-designed catastrophe-aversion system, however, regulators would anticipate employing this strategy, and before errors actually emerged, they would structure the regulatory system so that these errors would receive immediate attention for corrective action. On this score, our current efforts to deal with potentially catastrophic technologies are not sufficient. How we might improve the strategy of learning actively from error is discussed further in chapter 8.

Strategy 5
Set Priorities

Priority setting is a fifth strategy that works interactively with the strategies of testing and learning from experience. In the cases reviewed in this volume, the possible risks were so numerous and varied that it was impossible to evaluate all of them at once. Regulators had to set priorities for which risks to study, and at any given time, they focused attention on only small subsets of the possible hazards. This strategy provided a framework for testing and monitoring experience.

134

The most formal and explicit priority-setting strategy has been used in the toxic chemicals case. For existing chemicals, the Interagency Testing Committee explicitly designates the few chemicals each year that are most in need of additional testing. In the process of regulating new chemicals through the premanufacture notification system, whole classes of less dangerous chemicals are exempted from regulation. Attention can thereby be focused on classes of chemicals that pose a greater threat. For all types of chemicals, EPA uses three criteria to help quickly set priorities: production volume, structural similarity to known carcinogens, and exposure patterns. It is unlikely that these criteria will catch every danger. But considering the alternative being overwhelmed by the number of possible dangers priority setting is by far the lesser evil.

In the case of rDNA research, initial testing of possible risks focused on worst-case scenarios and on E. coli K-12, the most commonly used host organism for rDNA experiments. Decision makers at least implicitly made it a priority to study the gravest potential dangers in the greatest number of experiments.

As is true for learning from experience, there is considerable room for improvement in how regulators of risky technologies set priorities for testing and monitoring. Our analysis of the greenhouse case, for example, demonstrated the need for more formal priority setting to identify the crucial uncertainties.

In the regulation of nuclear power, attention has shifted from one issue to the next in reaction to events rather than as a result of any deliberate priority setting. Among other difficulties, this can result in a preoccupation with less important issues. For example, in a critique of current practices of the Nuclear Regulatory Commission, political scientist Aaron Wildavsky recommends that the NRC establish meaningful priorities by limiting the number of design changes for nuclear plants already in operation or under construction. Rather than forcing all nuclear plants to conform to the state-of-the-art, Wildavsky argues that more effective regulation could be achieved by requiring only those changes "deemed essential to meet performance standards."^[4] At present, the

135

NRC has so many regulations and design changes that the important ones become confused with the less important ones, and monitoring of key performance aspects becomes extraordinarily difficult.

The Complete Catastrophe-Aversion System

These five strategies for coping with the potential for catastrophe jointly compose a complete, integrated system:

1. Protect against the possible hazard; do so conservatively (strategies 1 and 2).

2. Reduce uncertainty; do so through prioritized testing and prioritized monitoring of experience (strategies 3, 4, and 5).

3. As uncertainty is reduced and more is learned about the nature of the risk, revise the original precautions: strengthen them if new risks are discovered or if the risks appear to be worse than initially feared; weaken them if the reverse proves true.

None of the cases in this volume has completely followed this idealized system. The monitoring and regulatory schemes for the particular risk in each case were strong on some points and weak on others. Of the regulatory approaches reviewed here, the one devised for rDNA research most closely approximates a complete catastrophe-aversion system.

As we have mentioned, some critics believe that regulators and the scientific community were too quick to discount the risks associated with rDNA research. But from a purely procedural perspective, the rDNA case comes very close to the ideal for handling technologies that present a potential for catastrophe. In retrospect, since this hazard is more containable and testable than those associated with the other technologies, rDNA research was the easiest problem to deal with. Nevertheless, the rDNA regulatory system provides a model of how society should cope with a high degree of uncertainty about risks combined with the potential for catastrophe.

136

Protective action was taken against the potential hazard of rDNA research by prohibiting the most risky experiments, rating all others according to degree of risk, and requiring prevention and containment measures based on the degree of riskiness. Uncertainty was reduced by learning from experience and through a deliberate program of risk assessment, including a number of worst-case scenario experiments. As uncertainty was reduced, the guidelines and prohibitions were gradually and sequentially adjusted.

While the exact mix of strategies appropriate in a given case obviously depends on the nature of the particular problem, the catastrophe-aversion strategy outlined above should be applicable to virtually any risky technology. Even without a clear perception that such a repertoire of strategies was evolving, society has been using these catastrophe-aversion measures. With an increased appreciation of the options, more systematic application of these strategies should be well within reach. Among other advantages, partisans and policy analysts attempting to map regulatory options will have a far more systematic framework within which to operate.

The catastrophe-aversion system formulated here is relatively simple, moreover, so there is a chance that it can be diffused gradually to a wide audience. It has not been our subject here, but the need for better public and media understanding of risky technologies is a widely shared belief among risk professionals.^[5] Such understanding would be valuable in itself for easing public anxiety where it is excessive and for increasing concern about some risks that now are receiving too little emphasis. Such improvements in the perspectives on risk management held by the media and the general public eventually should result in better allocation of governmental concern and risk-abatement expenditures.

Chapter 8 considers ways of improving the application of the catastrophe-aversion system. The remainder of this chapter attempts to distill the implications of our cases for professional thought about decision making under uncertainty and related topics. It is intended especially for social scientists; some readers may wish to skip directly to the concluding chapter, and can do so without losing the thread of the argument.

Implications for Social Science

Could contemporary theories of decision making have predicted what we would find in our five case studies? Not in sufficient detail to be interesting or useful, we believe. Could contemporary scholarship offer a rich set of recommendations about how to improve the strategies available for regulating risky technologies? Again, we find the relevant literature lacking. The cases examined in this volume suggest that the practice of decision making has advanced beyond available theory. What reassessment would enable theory to catch up to practice?

Analytic versus Strategic Decision Making

Scholarship on decision making tends to divide into two approaches: analytic and strategic.^[6] Using the analytic approach, a decision maker attempts to maximize the "expected value" of a choice.^[7] He or she must make an exhaustive search for alternatives, identify the consequences of each alternative, and predict the likelihood of each of the consequences of each alternative.^[8] Unfortunately, these requirements are impossibly demanding for any but the most simple of decision problems. They require precisely the conditions that most decision makers are denied: unambiguous information (to define the problem and analyze alternative solutions); time, money, and control over the environment (to enable the search for alternative solutions and the analysis of consequences); powerful causal models (to aid in analyzing consequences); and a complete, consistent way to order preferences (to estimate the relative value of the various consequences).

As an alternative to the analytic model, Simon, Lindblom, and others have proffered a set of decision theories we refer to as the "strategic approach." These include Simon's model of general problem solving, Lindblom's disjointed incrementalism and partisan mutual adjustment, March and Simon's approach to decision making in formal organizations, March's "garbage can" model, Steinbruner's "cybernetic paradigm," Etzioni's "mixed scanning" perspective, Dror's effort to synthesize the disjointed-incremental and rational models, and

138

other theories.^[9] These approaches to decision making differ in their description and prescription of search procedures, modes and means of analysis, decision rules, and preference structures. But all begin with the premise that decision makers face complex problems with uncertain information, inadequate resources, and ambiguous and sometimes conflicting values. All take as their central thesis that decision makers respond to these unhappy conditions by monitoring feedback from their choices and then adjusting those choices accordingly. All of these approaches are elaborate variations on a trial-and-error strategy.

The type of decision making apparent in our cases does not entirely fit either the analytic or the strategic approach but is clearly much closer to the latter. The decision makers in these cases exhibited a more deliberate and evolved form of the strategic model than the literature predicted. While these decision makers did employ certain elements of the analytic approach, it was typically in support of strategy rather than in its stead.

Matching Strategy to Problem

Our decision makers were most like the strategic and least like the analytic type in their orientation toward learning from error the sine qua non of the strategic approach. The underlying logic of the catastrophe-aversion system is to allow decision makers to learn, in time and with experience, more about the nature of the hazard and then evolve the necessary responses to it. This requires taking initial precautions, being conservative in the face of uncertain and potentially grave risks, and enhancing these approaches with testing, monitoring experience, and priority setting.

If there is a difference between our cases and the literature on strategic decision making, it is that in our cases there is more orientation toward learning as an explicit and deliberate part of decision making than is implied by existing theory. Lindblom's decision makers, for example, do not need to be aware that they are pursuing a strategy of serial adjustment to error; perhaps they even need to not be aware of it. The

139

"intelligence" of democracy is that diverse participants need only be concerned with pursuing their own partisan interests in order for serial adjustments of error (and gradual improvement of policy) to occur. Likewise, Simon's individual decision makers need not know that they are being boundedly rational. The constraints on their actions are determined by factors such as organizational structure, procedures, and training; they never really need to be aware that their calculations, deliberations, and actions are being constrained (in organizationally rational directions).

In contrast, our cases reveal that decision makers can deliberately adjust their strategies. They still rely on learning from error, but because the consequences of error are so much more severe than for ordinary problems, these decision makers cannot afford the luxury of the traditional approach of waiting for errors to emerge before undertaking corrective action. Decision makers implicitly, and sometimes explicitly, attempt to create conditions that would lead to learning from error conditions that would protect them from the worst consequences of error while at the same time allowing them to learn from those errors.

Because of the differences in the problems encountered, regulating risky technologies required substantial and deliberate variations in strategy from one technology to the next, and therefore, decision makers deliberately adjusted their strategies in accordance with the nature of the problem. This is best illustrated in the variations we discovered in the first of the five strategies, initial protections against possible hazards: the tactics to regulate toxic substances, nuclear power, rDNA research, and atmospheric threats were all different.

In each of these cases, then, there is a heightened or more advanced form of the strategic model in which the trial-and-error process emerges as a variable under the decision maker's control. The goal is to create a set of conditions that will allow decision makers to proceed through serial adjustment to error while simultaneously protecting society from the potentially harmful consequences of error. Decision makers' tactics vary with the nature of the risk. Decision making thus becomes a partly deliberate process of matching the strategy to the problem.

What Role for Analysis?

Even though our cases do not reflect the aspirations for rigorous analysis advocated by the analytic approach to decision making, analysis nonetheless plays a prominent role in regulating risky technologies. But the role of analysis is not explained adequately by either analytic or strategic theorists of decision making. We suggest that analysis is most appropriate when it is used in support of strategy .

Use of analysis in support of strategy is perhaps best illustrated in the rDNA case. First, measures were taken to protect against the potential hazard, then tests were run to determine whether the most severe of the presumed hazards were real. When these tests proved negative, the protective precautions were relaxed somewhat, and new tests were made on the remaining presumed hazards. Once again the safety precautions were adjusted according to the results of the tests. More tests on still other presumed hazards were made and were followed by further adjustments. The individual tests were not intended to prove rDNA research safe or unsafe, rather they were designed to provide specific data that could be used to narrow critical uncertainties. The accumulated clarifications allowed informed judgments concerning whether to tighten or loosen, at the margins, the tactics that had been deployed in support of an overall regulatory strategy. Analysis was extraordinarily important in this process, but it was integrated with and directed by a set of regulatory strategies. Analysis was not an alternative to strategy.

In the case of nuclear regulation, in contrast, nuclear advocates seemed at times as if they wanted to substitute analysis for a crucial part of the regulatory strategy. During the 1970s after the scaleup in reactor sizes required an emphasis on prevention, some regulators and activists advocated the use of probabilistic risk assessment for calculating absolute levels of reactor safety. But other professionals argued that such risk assessments were not well suited for measuring absolute levels of risk; rather they were useful for identifying weak links in reactor designs, thus indicating where wider margins for error or other tactics were needed to improve relative safety. Such

141

analyses help set priorities for focusing additional attention; but they could not reliably be used to determine the absolute safety of a reactor (that is, how likely an accident would be and how much damage it would cause).^[10] In other words, probabilistic risk assessment is an analytic tool to be used in support of strategy, not in place of it. But the NRC had not fully learned this lesson as late as 1985.^[11]

Analysis likewise threatens to overwhelm strategy in the greenhouse case. As discussed in chapter 6, attempts to reduce uncertainty about the timing and severity of the problem have been the dominant activity. This may be appropriate given the ambiguities about when climate changes will begin, the marked uncertainty about the effects, and the costs of action. However, the analysis is not being conducted strategically. It suffers from a lack of priority setting. Insufficient attention has been given to identifying and focusing research on the key uncertainties that would be important for policy makers. Moreover, the research focuses too exclusively on atmospheric science and oceanography, with little analysis of options for coping with or avoiding climate changes.

These cases, then, show analysis used in support of strategy, but they also indicate that such use of analysis is not yet widely understood or consistently applied. The regulation of risky technologies is handicapped by efforts to use analysis for inappropriate tasks and by failures to use analysis where it could be extraordinarily helpful.^[12]

Delayed Feedback

In the lower half of the matrix, one type of obstacle occurs when there is a time lag between policy and consequence. Obviously, serial adjustment to error is inappropriate if errors in policy do not become apparent for long periods of time. The most extreme example of such a problem is nuclear waste disposal where some consequences might not be apparent for hundreds or even thousands of years.

Even time lags of only a few years can sometimes block the normal process of serial adjustment to error. The Manhattan Project undertaken during World War II is a classic example. Speed in developing the bomb was considered essential during the project, partly because policy makers feared that the Germans were making rapid progress in developing their own bomb. The key element in developing the bomb was the production of enriched uranium. The problem was that each of the alternative approaches for enriching uranium was fraught with uncertainties. Because of the emphasis on speed, decision makers could not afford to rely on ordinary trial and error; they did not have the time to try one approach, wait for feedback, and then change approaches. Their response to this dilemma was to modify the basic strategy: they pursued several alternative approaches simultaneously and then made adjustments as feedback on each emerged.

This simultaneous trials strategy was employed again in the early 1950s in the development of the nuclear submarine and again in the late 1950s in the development of nuclear power reactors. This strategy is now common in industrial research and development when decision makers cannot afford to wait for feedback. Some similar adjustment in decision strategy must be made whenever delayed feedback is expected to interfere with normal trial and error.

Unclear Feedback

Feedback also can be problematic when the causal links between trial and error are obscure or complex. Imagine that a police department changes its crime fighting tactics for instance, it puts more officers on beats in the subways or more patrol cars in high crime districts. Subsequently, crime rates decline. Should we infer from this feedback that the change in tactics succeeded? Possibly, but the feedback could result from other changes. Even if we set aside questions about the reliability of crime statistics, the decline in crime rate could be due to changes in the economy, other social programs, or even the weather. Alternatively, suppose the crime rate increased. The new tactics might be judged a failure when in fact the changes might actually have had a positive effect that was offset by countervailing changes in other variables. This would be enough of a problem, but if this phenomenon was not recognized, spurious conclusions could be drawn. If decision makers mistakenly learn from experience about a tactic that reduces crime, they may apply the same tactic to different situations only to find to their surprise that it does not work.^[14]

How common are problems in which the causal links between trial and error are unclear? LaPorte et al. suggest that many social problems that are treated in relative isolation from one another are, in fact, definitely interconnected. They might be thought of as "semi-lattices," rather than the nearly decomposed or independent systems often assumed by decision theorists.^[15] In the social policy arena poverty, education, crime we suspect that unclear causal links may be more the rule than the exception, but there has not been enough analysis of this matter to reach firm conclusions.^[16] Our point is that normal trial and error implies relatively straightforward causal links between policy choices and subsequent feedback, and on many occasions these links are unclear.

Is there a strategy for modifying normal trial and error that can be used for these cases? The answer is unclear. One approach has been, in Wildavsky's terms, a "strategic retreat from objectives."^[17] Rather than continue to try to solve social problems made difficult by these extensive interconnections, an alternative is to pursue more modest objectives. In the criminal justice arena, for instance, some professionals have

146

retreated from the goal of rehabilitating criminals; the revised objective is merely to take the offender off the streets for a while and make it clear that crime results in punishment.

While strategic retreat is a practical response, it is, in effect, an admission of failure. Since multiply-interconnected problems cannot be solved, we retreat to problems that are more amenable to solution. Is there an alternative to admitting failure when cause and effect are too confused for normal trial and error? Since the problem is that extraneous variables intervene in the cause-effect sequence, the solution is to control these variables. One approach is through so-called quasi-experiments that ordinarily entail a strategically selected pilot program or multiple pilot programs. Just as scientific experiments represent a form of very tightly controlled and monitored trial and error, so pilot programs represent trial and error under quasi-controlled conditions.

Unfortunately, the history of quasi-experimentation has been fraught with political obstacles.^[18] Even when such obstacles are overcome, as in the Income Maintenance experiments of the 1970s, the results of a pilot program may be ambiguous. Despite these limitations, quasi-experimentation appears to be the only sensible strategic response (other than strategic retreat) to unclear causal links between trial and error.

Problematic Feedback and Severe Consequences

This brings us to the worst of both worlds, where decision makers face problems on which feedback is unavailable and the potential consequences of error are catastrophic. The archetypal example of this situation is crisis decision making. In this type of decision making, information is very scarce, time very short, and the margin for error very narrow.^[19] We do not know much about what kinds of decision strategies are appropriate under these conditions. Trial and error is obviously inappropriate; decision makers cannot afford to err, and they do not have the time to wait for feedback.

Thus, the temptation in such cases is to fall back on the analytic approach: if decision makers cannot rely on serial adjustment to error, then they must be as rigorously analytic as possible identifying objectives, canvasing alternatives, and

147

reviewing information about the possible consequences. Janis and Mann's study of decision making under stress comes close to this prescription.^[20] The problem with the analysis-dependent approach is that it assumes the very conditions that decision makers do not have. In crisis decision making, objectives are often ambiguous, alternatives severely constrained, and information about consequences little more than guesswork. However, if decision makers in crises cannot rely on analysis and cannot proceed by trial and error, how should they proceed? This is the dilemma.

There is at least one arena in which strategies have evolved for coping with crisis decision making pilot and air traffic controller training. An air emergency has all the elements of decision making under crisis: very little time to act, few alternatives, and virtually no margin for error. Pilots and controllers are trained for such crises through simulation exercises. The nuclear industry now is beginning to employ the same techniques for training nuclear power plant operators. These operators will learn how to act in emergencies through simulated trial and error. War games and simulated nuclear attack exercises serve much the same function. Decision makers cannot proceed by trial and error during the real event, so they train for it by learning from simulated events.

While it may be unrealistic to expect a president and his top advisors to subject themselves to simulated crises, perhaps their immediate subordinates should regularly prepare for future crises through simulation exercises and reviews of past crises. If this training was done properly (and this would be no small task), it would help decision makers to identify weak links, likely trouble spots, useful delay tactics, and fallback positions. These skills could prove invaluable in a real crisis.

Conclusion

The combined categories of problem types discussed here are depicted in Figure 5. While too simplified to be anything but suggestive, this representation suggests three conclusions.^[21] First, it seems possible to analyze the character of problems; a more in-depth study of problems would need to

148

[Full Size]

Figure 5.
Types of decision problems

include more dimensions than just consequences and feedback patterns, and it would need to focus on particular problems rather than the combined categories we show.^[22] Second, as we evaluated one type of problem after another, we observed that different decision-making strategies emerged; much more investigation is necessary to uncover and specify the complete repertoire of strategies and to link them empirically and normatively with various problem types.

Third, although even sketchier, Figure 5 suggests a way of thinking about the strategies discovered in this volume and of responding more generally to problems that fall outside the upper left quadrant of the matrix. The impulse is to abandon trial and error and to pursue a more analytic approach to such problems. (Technology assessment and risk-benefit analysis are prime examples.) But, as stated, analysis when pursued in isolation from strategy is inappropriate because it requires the very conditions that decision makers do not have.

Instead of abandoning trial and error, the more appropriate course is to artificially create conditions that make serial adjustments possible. In effect, this means restructuring the problem so that it can be handled more like a normal policy

149

issue. If the potential consequences of error are severe, steps should be taken to protect against these consequences, and then decision makers should proceed by trial and error. If the causal links are unclear, steps should be taken to control the intervening variables, and decision makers should then proceed by trial and error. If time lag between trial and error is too long, decision makers should implement several alternatives simultaneously and then proceed by trial and error. In short, if the decision problem is inappropriate for trial and error, decision makers should create conditions that make it more appropriate.

In general, our analysis suggests the need for social scientists to pay more attention to variations in the nature of social problems, to variations in the nature of decision strategies, and to ways of matching strategies to problems.