IV. A DECISION-BASED APPROACH

As we have discussed above, the conventional approach offers a useful foundation from which to start developing a monitoring plan, but the approach can be thwarted by some of the complexities involved in monitoring a large-scale, complex set of activities such as those stipulated through the ICBEMP or NWFP. Two obvious shortcomings of the conventional approach are (1) the monitoring effort is not directly tied to the decision process itself, rather leaving this important feedback loop to a more ad hoc process, and (2) the conventional approach does not explicitly address how one prioritizes among the multiple possible indicators and questions that might be monitored.

In this section, we reach beyond the conventional approach to bring in concepts of statistical decision theory and adaptive management. Our purpose is to introduce a more rigorous framework for tying monitoring to information needs for decision making. Specifically, we propose a formalized method for hypothesis formulation and testing embedded in a decision analysis framework. The methods that we introduce are not novel or untested, enjoying extensive application throughout industry and government, but seemingly have not seen widespread exposure or use within the FS or BLM. Because the ideas are new to many, our discussions are relatively elementary and the examples simple. This does not suggest that ICBEMP or NWFP issues are too complex to yield to similar analyses, though the effort to complete such analyses will be substantial.

 
A. Statistical Evidence

We begin with a discussion of the nature of statistical evidence, that is, how should one interpret data. In this task we will rely heavily on the concepts of likelihood, following the excellent description by Richard Royall (1997).

In the preface to his book, Statistical Evidence: A likelihood paradigm, Royall (1997) states,

Royall proceeds from there to note that all is not well in statistical science and builds his case for adopting the likelihood paradigm for statistical inquiry. We will not go into the details of the likelihood paradigm here, nor describe how it differs from classical statistical techniques, but it is worth noting that most of the statistical techniques used in ecology and elsewhere do not follow directly from the likelihood paradigm. Rather, they follow from statistical theory advocated by Neyman and Pearson (1933) (e.g., hypothesis testing, Type 1 and Type 2 error probabilities), or alternatively, Fisher (1959) (e.g., significance tests). There is ongoing debate among the statistical community as to whether the likelihood paradigm should play a more central role. What is not in debate, however, is that the likelihood paradigm is fully consistent with a statistical decision making framework.

What is the likelihood paradigm, and what role can it play in linking monitoring and decision making? We start, following Royall, with three basic questions that one can answer when confronted with data: (1) what do the data say with regard to the relative truthfulness of a given hypothesis or set of hypotheses; (2) what should one believe given the data; and (3) what action should one take.

The first question is one that falls solidly within the realm of science. That is, there should be formal methods for determining which hypothesis is best supported by observed data, and further quantifying the strength of that support. The law of likelihood quantifies this evidentiary support. In simplest terms, the law of likelihood states that the hypothesis which is best supported by the observed data is the hypothesis which has the highest likelihood of having produced that data. Furthermore, the strength of evidence supporting a given hypothesis vis-à-vis a competing hypothesis is determined by the ratio of their likelihoods. For example, if the likelihood of observing X = 3 is 0.2 under hypothesis A and 0.8 under hypothesis B, and X = 3 is observed, then B is 4 times more likely to be correct than A, given the observation.

The second question ventures beyond the data themselves and into the realm of personal belief. What one believes given new data will depend in part on prior beliefs. For example, if a tossed coin lands on heads, is that sufficient evidence for one to believe that every toss will be heads? What if three tosses in succession land on heads? What if three hundred tosses land on heads? In each case, the most likely hypothesis given only the observed data is that all tosses land on heads. Yet, our prior belief that it is a fair coin leads us to continue to believe that a toss landing on tails is equally likely, despite the occurrence of one heads or three heads in succession. Even the most die-hard believer in a fair coin, however, would have her belief shaken by 300 successive heads with no tails.

This intuitive sense of how evidence affects beliefs was first articulated mathematically by Reverend Thomas Bayes in 1763 and has given rise to a field of statistics known as Bayesian statistics. Bayesian statistics allows prior beliefs to be combined with empirical evidence in order to update beliefs regarding parameters (Press 1989), taking advantage of Bayes theorem and the likelihood principle. Bayesian approaches are common in the decision analysis literature and have been suggested for ecological analyses (Ellison 1996), but remain heresy to many empirically based statisticians because of the subjective nature of prior beliefs (see Dennis 1996). (Other classical methods have subjective elements in them as well, but users don't readily admit to it.)

The last question of what one should do moves beyond evidence and belief and into the arena of risk management and values. Consider the coin tossing example again. In this case, imagine that someone extracts a coin from their pocket and tosses it five times in succession, and each time it lands on heads. Before tossing it a sixth time, however, a bet is proposed. If the coin lands on tails, the tosser will pay you $1000; if it lands on heads, you pay him $100 dollars. Do you take the bet? The answer, of course, depends on a variety of factors, among them the character of the person tossing the coin, i.e., do you believe that the coin tosser is intentionally trying to deceive you. Assuming for the moment that the tosser is not a statistician or other scoundrel, foremost in one's consideration of the bet is a personal attitude towards risk. For some, $100 is valued too highly to risk it in all but the surest cases (i.e., they are risk adverse). For others, $100 seems a small sum to risk for a possible 10-fold payoff (i.e., they are risk-neutral); there are even others that will be excited by the gamble and take the bet irregardless of the odds (i.e., risk seeking; casinos love these individuals). Both risk-adverse and risk-neutral individuals will want to carefully assess their belief that the coin is fair; each group, however, will accept or reject the bet at a different odds ratio.

Examining evidence, beliefs, potential payoffs, as well as attitudes about risk falls within the domain of statistical decision theory (Clemen 1996; Hamburg 1987). In the following sections, we present a more formal representation of these ideas and show how decision analysis can be used to both inform decisions and structure a monitoring program. By decision analysis we mean a rational, formal analysis of options available in the course of making any decision, and the potential consequences of those options. We are not referring to any prescribed step in the NEPA process (for those handicapped by an ingrained sense of FS-speak). For a brief overview of the lexicon of decision analysis, see the paper by Tom Spradlin (1997) on the WebPages of the Decision Analysis Society (http://www.fuqua.duke.edu/faculty/daweb/lexicon.htm).

B. Influence Diagrams

The next step in building a framework for linking monitoring to decisions is to introduce the concept of influence diagrams (Howard and Matheson 1981). The purpose of an influence diagram is to represent the decision process in a way that explicitly recognizes the uncertainty in consequences or outcomes of the decision. Influence diagrams consist of nodes or variables connected by directed arrows. There are three kinds of nodes: (1) decision nodes representing alternative actions that can be taken by decision makers; (2) chance nodes representing events or system variables that are outcomes of the decision or other chance variables; and (3) value or utility nodes, variables that summarize the final outcome of a decision. In business decisions, value nodes are often expressed in monetary units. For other kinds of outcomes, the relative satisfaction offered by a particular outcome is summarized by its utility, a non-dimensional metric that allow comparisons of dissimilar elements (e.g., fish versus timber). Relations between outcomes and utility are expressed as utility or preference functions; such functions reflect both comparative value and attitudes about risk (Keeney and Raiffa 1976). The numeric values of the preference function generally is less important in ranking alternatives than the function's shape. This feature helps focus discussion of preference functions, which can be a contentious matter among interested parties with very different value systems. While it is possible to analyze decisions without explicitly assigning values or utilities to outcomes, the act of choosing one outcome or the other as preferable implicitly reveals a preference function.

The arrows connecting the nodes in an influence diagram represent conditional dependence. That is, the likelihood of the receiving node taking on a particular value depends on the decision or the realized values of donor chance nodes. Utility node are strictly receiver nodes; they assume particular values based solely on the realized value(s) of chance nodes, i.e., the outcome. In the parlance of decision analysis, the utility values are attributed back to the decision. Thus, the objective of the influence diagram is to identify the decision option with the highest expected utility. As we will discuss below, influence diagrams serve other purposes as well.

The key to making an influence diagram more than simply a schematic representation of the interaction of decisions and chance variables is in the mathematical representation of causal dependencies. The strength of these dependencies is captured in conditional probability matrices that link chance nodes to decisions or other chance nodes. The underlying framework follows probability theory and the likelihood principle. Influences are propagated through the network using Bayes theorem. This mathematical framework provides a strong conceptual link between influence diagrams and statistics that is absent in other modeling approaches (e.g., simulation modeling, fuzzy logic). It also provides a natural link back to how one uses data to update or verify the diagrams.

An Example using Wildfire and Fish

We illustrate the use of an influence diagram with an example involving wildfire and fish populations. At issue is a decision regarding the level of fuels treatment (thinning, prescribed fire, etc.) that might be called for within a watershed with unnaturally high fuel loads. The fuel loads suggest a relatively high likelihood of a stand-replacing wildfire occurring within the next 50 years. The decision is complicated by the fact that the watershed supports a sensitive fish population. There are two competing lines of reasoning regarding the fuels treatment. The first argues that a high intensity, stand-replacing fire leads to loss of scenic, recreational, and other values, and furthermore could lead to increased flooding and erosion, and thus degradation of instream fish habitats. The competing argument is that while fire effects are uncertain, the fuel treatment itself would surely increase erosion to some extent, which might also affect fish. The worst possible scenario would be fuel treatment that was ineffective in preventing a high intensity fire, but which aggravated erosion and habitat degradation even more once a fire occurred.

To examine this decision, we constructed the influence diagram shown in Figure 4.1. The diagram consists of a single decision node, six chance nodes, and a utility node. At the top of the diagram is the decision node, fuel treatment. Two possible fuels treatment levels are proposed, treat the entire watershed or treat the upslopes only and leave the riparian areas untreated; a third decision option is to do no fuels treatment (Figure 4.2). A chance node, fuel ignition, represents the frequency of natural fire ignitions and is the only independent chance node in the diagram, i.e., its value does not depend on other nodes but is determined exogenously. Fuel treatment and fire ignition combine to affect the likelihood of wildfires of different intensity within the watershed. Wildfire and fuel treatment in turn affect fire in the riparian zone; wildfire also affects flooding regime. Instream fish habitat is the next link in the network, and is affected by the combination of fuel treatment, fire in the riparian area, and the flooding regime. The terminal chance node, fish population, is affected only by instream habitat. Utility is expressed as a function of wildfire, instream habitat, and fish population. The reason for including fish habitat in the utility function is the recognition that fish populations can be affected by factors other than habitat that are not explicitly included in the diagram. Thus, some utility is given to an outcome of improved or protected habitat even when the fish populations fail to follow suit.

Each of the chance nodes is divided into two to four discrete levels. The relationships among them are quantified by the conditional probability matrices. The values in these matrices were contrived for this example and should not be taken seriously. They are available for inspection in Table 4.1, or by clicking on each node in Figure 4.1 (if we can figure out how to make this work). Table 4.1 also contains the utility values assigned to various outcomes. For example, one possible outcome would be no fires in the watershed, no fire in the riparian zone, no change in flooding frequency, no change in fish habitat, and a stable fish population. (When all possible outcomes are displayed as their own pathway, the resultant diagram is known as a decision tree.) When a decision is specified, its "influence" is propagated through the network of nodes, changing the probabilities of all possible outcomes. Overall expected utility is calculated by summing the utilities associated with each possible outcome, weighted by their respective probabilities, given a decision.

Based on the influence diagram as constructed, we can examine the relative utilities of different fuel treatments (Table 4.2). The results suggest that the optimal decision depends on the natural fire ignition frequency. If all possible frequencies (rare, infrequent, or frequent) are equally likely, the suggested preferable decision is to treat the entire watershed. Now suppose that fire ignition frequency can be accurately estimated; then the decision can be evaluated based on known ignition frequency. In this case we find that the preferable decision depends on the ignition frequency. If fires are ignited either rarely or frequently in the watershed, the preferable decision is to treat only the upslope areas. If fires are infrequently ignited, the preferable decision is to treat fuels in the entire watershed. Note that in our contrived example, the cost of treatment is not factored into the expected utility. In cases with many possible decision options, one can plot the expected utility versus costs and define a cost-effectiveness frontier, i.e., the set of alternatives that maximize utility for a given cost.

Table 4.2. Calculated expected utilities for different fuel treatment options, conditional on fire ignition frequency
 

C. Benefits of the Influence Diagram

As described above, influence diagrams are generally constructed for the purpose of identifying preferable decisions in an uncertain environment. This type of pre-decisional analysis, however, is but one benefit of having a formal influence diagram. Other benefits that are germane to the issue of developing a monitoring plan include the following.

1. The influence diagram is an explicit, formal representation of all knowledge of the system that is relevant to the decision. There is no vague logic trail behind the decision with competing world views arguing this or that is the best decision. All assumptions and preferences are fully revealed. In a sense, the influence diagram represents a formal, quantified, composite hypothesis of how the world operates, including how the decision makers value potential outcomes. Hypotheses are embedded in the influence diagram in two ways: (1) in the structure of the diagram itself; and (2) in the conditional probabilities assigned to each chance node. In a more complete analysis, one can construct alternative influence diagrams that reflect different views or hypotheses of how the world operates. The decision analyst can then seek out a decision that is optimal across all competing hypotheses. From a monitoring perspective, the hypotheses embedded in the influence diagram can be examined in a scientifically rigorous manner using empirical data. We return to this point below.

2. The mathematical structure underlying the influence diagram lends itself to quantitative sensitivity analysis. Thus, one can readily determine where uncertainty in chance nodes is most influential in affecting outcomes, and the degree to which the conditional probabilities are supported by data. Alternatively, one can also readily discover the sensitivity to the utility function. Both insights are extremely valuable. The first points to areas of needed ecological research or priorities for ecological monitoring to either reduce the uncertainty in a chance node (if possible) or refine the node description and conditional probabilities through data collection and analysis. The influence of the utility function suggests an active role for socioeconomic research to accurately determine how the public and decision makers value resources and regard risk.

3. Influence diagrams are a natural mechanism for integrating scientific information across disciplines. For example, if the fire and fish diagram that we illustrated above were to be constructed with the best available information, the services of a fire ecologist, a hydrologist, a fish biologist, and a social scientist would be required to build and parameterize the model. Add to that a decision analyst if none of the others have the prerequisite skill to put the influence diagram together. These sorts of interdisciplinary teams are not new to the FS or BLM; what is new is that there rarely seems to be a common framework in which to express interdisciplinary understanding of the system of interest.

4. The modular structure of the influence diagrams allows them to be combined or partitioned in different ways depending on the scale and complexity of the decision in question. Furthermore, relations between nodes can also be collapsed or expanded depending on the detail required. Going back to our fire and fish example, the effects of riparian fire and flooding regime on instream habitat are complex. Our example simplifies this relationship tremendously in order to capture no more than is necessary to inform the decision at hand. One might want to expand this segment of the model in order to more fully articulate relationships. Such expansion might lead to improved understanding that could be used to update the simpler version, or to create an alternative structure that better distinguishes conditions under which fire in the riparian zone has positive or negative benefits.

5. The influence diagram approach cleanly distinguishes how one addresses the three question posed above from Royall (1997). While the objective of the model is to answer the question, what does one do, there are more specific questions as well. Scientists can play an objective role in structuring and parameterizing the diagrams in a way that is most consistent with scientific understanding and data, answering the question of which hypothesis is best supported by data. Similarly there are formal statistical techniques for updating these diagrams as monitoring data are collected, using Bayes theorem and the likelihood principle. The decision analyst can decide, in combination with the decision makers, if there is sufficient empirical data to build the diagrams or if more subjective professional opinion is required, quantifying beliefs in the process. If necessary, there are formal methods for eliciting such information (Morgan and Henrion 1990). Both decision analysts and social scientists will be engaged in building utility functions.

6. Influence diagrams are a natural fit to the concept of adaptive management. Building models such as influence diagrams that explicitly acknowledge uncertainty is a central step in adaptive management (Walters 1986). There are two kinds of uncertainties contained in an influence diagram: those arising because of random events in nature, and those arising from failure to fully understand how the world operates (ignorance). By examining these models, one can tell not only which uncertainties are likely to impede management objectives, but also identify management decisions that if properly monitored can reduce uncertainties based on ignorance. Such insights can lead to management actions for the sake of learning. That learning experience can be formally captured in decision models, which in turn improves management efficiency.

Table of Contents | Next Section | References