The following questions were raised by various individuals concerning the September 8 draft of this document. Each question is set off by a >. Answers provided by Danny Lee are inserted within or follow each question.

Paul Hessburg (PNW, Wenatchee) asks:

> Can such belief networks be built using fuzzy sets vs. probability matrices?
> Assuming yes, do they function pretty much in the same way?

Fuzzy sets and belief networks (the foundation of inference diagrams) are
somewhat similar, but yet fundamentally different. In
a belief network, one has nodes that can assume a range of values. There is an
inherent assumption that these values are mutually exclusive, and that the
"real" value can be determined unambiguously. The uncertainty arises from
basic ignorance as to what is the true value. In our fire-fish example, fire
ignition was split into 3 levels: rare, infrequent, and frequent. It is
assumed that if you know the fire ignition frequency (ignitions/year), you can
readily classify the watershed into one of these three classes. Fuzzy logic
takes a different approach and assumes that we can measure a quantity
accurately, we're just not sure what it means. So in the fire ignition
example, a ignition frequency of 0.2 fires per year might be classified
somewhere between rare and infrequent. As far as I can tell (and I'm no expert
on fuzzy logic), there's no parallel in fuzzy logic for ignorance (i.e., you'd
know it if you saw it, but you can't see it)., or for personal belief (e.g.,
I've not seen it, but I'll lay odds that it's such and such). Fuzzy logic is
great for taking concrete observations and translating them into more abstract
propositions. That is, we might measure vegetation composition and structure
and translate that into a proposition about "forest health".

There is a very real difference between fuzzy logic and belief networks in how they
use or interpret data. Fuzzy logic is a formal method of interpretation that is
designed to capture how people reason. With new data, you get a new interpretation
based on the fuzzy logic that you have constructed. This logic is not amenable to
validation, given data. A fuzzy logic trail is inherently arbitrary, ergo, all such logic
paths are assumed to be correct. In contrast, a belief network can be viewed as a
hypothesis that can be formally compared to data using the likelihood principle, and the
underlying conditional probability matrices can be updated with new data using
Bayes theorem. To me, this tie to statistical methods makes inference diagrams a
more coherent and defensible paradigm to drive development of a monitoring program.
There may be opportunities (and reasons) to link to fuzzy logic rules (e.g., for subjective
statements about resource conditions), but the tough and dirty work of drawing inferences
from data should be left to the more statistically based inference diagrams.

> When building influence diagrams, scientists will surely disagree to a large
> extent on some scientific matters that reflect how we think the world works.
> This would surely effect what models predict are most critical outcomes and
> most key features to monitor. How do decision analysts gauge the effect of
> this rather chronic problem of ignorance or scientific disagreement on decision
> models and decision-making?

They will disagree, no question about that. Our argument is that at least we'll
be able to see exactly where they disagree and to what extent it actually makes
a difference in the decision. Furthermore, one can carefully craft experiments
and monitoring designs to resolve the disagreements on a level playing field,
rather than allowing the competing groups to dash off and find their own
supporting evidence for their particular view of the world. This is an area
where we need to more fully develop an example. Decision analysts address these
kinds of disagreement the same as any other objective scientist, by trying to
develop a "weight of evidence" that will support one view over the other.
 

Ralph Heft (BLM) asked:

> If I understand what I read, which is a wild assumption, the decisions
> anticipated from the Project, especially in aquatics, don't appear to me to
> be compatible with your proposed form of analysis. Wouldn't this form of
> analysis need alternative forms of management to test. The decisions
> coming out of the project are mostly of the one size fits all variety, e.g.
> RCAs. I can see how the traditional monitoring and evaluation could tell
> us if we are meeting our objective but I don't understand how a
> comparison of management practices could be done. Is your proposal
> more applicable to validity monitoring than effectiveness monitoring?
>

We're all guessing as to what the final ROD will look like at this point, but
it's pretty certain that no direction can be truly "one size fits all" for a
large, heterogeneous landscape like the interior CRB. My sense is that the ROD
will provide general direction that will require a series of management
"choices" at various levels through the implementation process. The same or
similar management direction applied in multiple basins with different
topography, vegetation, previous disturbance histories, and agency budgets can
result in very different outcomes. These different outcomes, if properly
monitored, can tell us something about how systems behave and whether we are
making the right choices to efficiently meet our objectives. You're right if
you're suggesting that experimental management can help us learn more
efficiently--that's the whole premise of adaptive management. But simply
implementing a single strategy can also be instructive.

As to the validation versus effectiveness question, I would argue that
effectiveness monitoring that is not properly integrated with validation
monitoring is essentially useless. What good is it to know that you are/are
not meeting an objective without knowing whether that progress or lack thereof
is directly tied to the management actions in place? It's like
blaming/praising the rooster for the sun rising. Furthermore, many of our
objectives will require decades before we can detect measurable progress. How
do we check our progress or refine our activities in the meantime without
validation monitoring? The framework that we propose can be used to seamlessly
tie effectiveness and validation monitoring together.

> What you are proposing appears to need lots of data which translates to
> lots of money. I'm use to evaluating decisions with available data,
> translated as not much, and going with the preponderance of the
> evidence what ever it is. I can't imagine this Project will result in
> Congress giving us gobs of new money for monitoring. How can your
> proposal be initiated within existing budgets? If it can't, what options are
> there for the best monitoring system we can afford?

An important point has been missed here. What we propose REQUIRES no more
data than any other approach. I can build fairly sophisticated decision models
with little data, or fairly simple models with lots of data. The
sophistication of the models does not depend on availability of data. What
does depend on data is the level of uncertainty within the models. When you
say you're used to "going with the preponderance of the evidence, whatever it
is," you're admitting to making decisions under uncertainty. What I suspect
you've not done is explicitly acknowledge or try to quantify the uncertainty in
your projections of the outcomes of your proposed actions, nor is there any way
to identify what piece of information would best reduce that uncertainty. You
can't answer the question, "what options are there for the best monitoring
system we can afford," without a formal analysis. That's the beauty of the
influence diagrams. We can do a cost-benefit analysis of different monitoring
options to see what information most efficiently tracks system behavior and
tells us whether progress is being made. If there are limited budgets, the
analysis tells you where to invest it.
 

Bruce Rieman (RMRS, Boise) asks:

> -although you make the point that monitoring without an explicit decision
> structure is not likely to work..... I still struggled to make the example link
> back to monitoring. Extension of the example to the selection of monitoring
> variables, approaches would help me... Intuitively the link is obvious but
> making that link explicit seems as difficult as formulating a formal decision
> framework.
>

This is something that clearly needs more explanation. We'll work more on the
example to show how sensitivity analysis can show where reducing uncertainty in
the chance nodes can contribute the most in providing confidence in the
outcome, and also show how monitoring data can be used to update an influence
diagram by altering the conditional probabilities. We left out many of the
details in the first draft, in part because of lack of time, but also because
we wanted to gauge the response to these ideas before we invested considerable
more effort expanding the examples.

> -Given that the guts of the paper are on the decision framework and the
> monitoring discussion is justification that got you there shouldn't the title
> reflect more the former than the latter??
>

Perhaps. We were charged with developing a conceptual framework for monitoring,
and we do include a discussion about the conventional approach that does not
explicitly involve decision analysis; so it seem appropriate to keep monitoring
in the title.

> -the real diagram for the example you propose is likely to be more complex (and
> contentious??). Obviously it would be desirable to explore the construction of
> such a diagram/model for real. It is not clear to me that we can move very far
> beyond uniform probability vectors or even identify the appropriate link
> functions for the very large scale problems, but maybe it does simplify more
> than I anticipate. The fire- fish example is a particularly
> relevant/contentious example that could draw a lot of attention/interest and
> support. It would be nice from a number of perspectives to pursue that.
>

It's hard to speculate what would happen in a real-world exercise. Model
building is an iterative exercise, and these kinds of models are quite easy to
expand or contract depending on the resolution desired. The key to building
influence diagrams is to make the model no more complicated than is necessary
to inform the decision. We don't need an explicit model of every possible
ecological interaction. I think we need a prototype application to see how it
would work out, and agree that the fire-fish issue would make an excellent test
case.