Dollars and Jens
Friday, September 26, 2008
 
Rational Choice in an Uncertain World
I can't sleep, so I thought I'd offer a couple excerpts of a book I'm reading. It's written by two psychologists on the subject of decision making.
At the risk of overgeneralization, we offer here some conclusions about typical judgment habits that are true of both amateur and expert judgment:
  1. Judges (even experts) tend to rely on relatively few cues (3 to 5). There are some exceptions to this generalization, ...
  2. Few judgment policies exhibit nonlinearity — again, contrary to many judges' own beliefs about their policies.
  3. Judges lack insight into their policies — they are unable to estimate their own relative "cue utilization weights" accurately, especially when they are expert and highly experienced.
  4. Many studies ... reveal large individual differences in types of policies ... and low interjudge agreement on the judgments themselves. ... At a minimum, interjudge disagreements tell us someone is wrong and undermine our confidence in all judgments.
  5. When associated, but undiagnostic, irrelevant information is presented to judges, they become more confident in the accuracy of their judgments, although true accuracy does not increase.
Several pages later:
For years the nagging thought kept recurring: Maybe any linear model outperforms the experts. The possibility seemed absurd, but when a research assistant had some free time, Dawes [one of the authors of the book] asked him to go to several data sources and to construct linear models with weights 'determined randomly except for sign." ... After the first 100 such models outperformed human clinical judges, Dawes constructed 20,000 such "random linear models" .... On average, the random linear models accounted for 150% more variance between criteria and predictions than did the holistic clinical evaluations of the trained judges. For mathematical reasons, unit weighting (i.e., each variable is standardized and weighted +1 or -1 depending on direction) provided even better accountablility, averaging 261% more variance.
Note, in particular, that the last figure requires that the experts were predicting no more than 27% of the variance in the relevant statistics.


Powered by Blogger