Metrics of effectiveness for micro and macro decision analysis applications

Abstract

By macro decision analysis applications we refer to systems supporting decisions that directly involve human agents whose decisions may have major impact on the agent's organization. By micro decision analysis applications we mean systems where large numbers of decisions are made by automated decision systems, but the effect of each individual decision is not significant. In the former, few trials are available for statistical evaluation, but diverse meaningful metrics may be established which taken together offer reasonable evidence that a given application is cost effective. In the latter, metrics to demonstrate an increase in decision performance may be demonstrated statistically, but an analyst often has little insight into the decision process. An application involving the acquisition of a super computer is used to illustrate a macro-decision application, while a neural-net based application elucidates the micro case. It is then argued that the ultimate acceptance of decision analysis technologies depends on the provision of both process and performance metrics.