This summer, I assigned Tetlock and Gardner’s Superforecasting (see here, here, and here) to my homeschoolers. They took to this masterpiece of political psychology like fish to water. I’d already taught them about human beings’ wacky mapping from language to quantitative probabilities, but these passages have really stuck with them:
In March 1951 National Intelligence Estimate (NIE) 29-51 was published. “Although it is impossible to determine which course of action the Kremlin is likely to adopt,” the report concluded, “we believe that the extent of [Eastern European] military and propaganda preparations indicate that an attack on Yugoslavia in 1951 should be considered a serious possibility.” …But a few days later, [Sherman] Kent was chatting with a senior State Department official who casually asked, “By the way, what did you people mean by the expression ‘serious possibility’? What kind of odds did you have in mind?” Kent said he was pessimistic. He felt the odds were about 65 to 35 in favor of an attack. The official was started. He and his colleagues had taken “serious possibility” to mean much lower odds.
Disturbed, Kent went back to his team. They had all agreed to use “serious possibility” in the NIE so Kent asked each person, in turn, what he thought it meant. One analyst said it meant odds of about 80 to 20, or four times more likely than not that there would be an invasion. Another thought it meant odds of 20 to 80 – exactly the opposite. Other answers were scattered between these extremes. Kent was floored.
Ideally, analysts would adopt standard conventions on the correct way to translate from English to math. But since that won’t happen anytime soon, my son Tristan proposed a simple, effective half-measure. Namely: every serious book (and perhaps every article) should include a Probability Glossary in the opening pages, alongside the List of Illustrations and List of Tables. In these glossaries, each author would explicitly state what probabilities (or probability ranges) he assigns to probability-relevant English words. For example, here are roughly the probabilities I have in mind when writing this blog:
English Term |
My Probability |
Absolutely certain |
100% |
Certain |
>99% |
Almost Certain |
≈95% |
Highly Likely |
>80% |
Probable |
>60% |
Likely |
>51% |
Toss-up |
45-55% |
Unlikely |
<49% |
Possible |
1-35% |
Highly Unlikely |
<15% |
Almost impossible |
≈5% |
Impossible |
<1% |
Absolutely impossible |
0% |
Filling out this table makes me self-conscious of the inadequacies of my current usage. But it’s still a big step forward in the War for Clarity. Indeed, reading this table might make you realize you and I suffer from illusory disagreement – or illusory agreement. And even if publishers are loathe to add a Probability Glossary, there’s no reason why every thinker who wants to signal seriousness couldn’t publicly post one on his blog or webpage.
When you do, please credit my son for the idea…
READER COMMENTS
Effem
Aug 29 2017 at 2:30pm
What about “beyond a reasonable doubt?”
Matthew Moore
Aug 29 2017 at 2:34pm
Interesting. I wonder how culture mediates. I would never use likely and unlikely in this way. For me, they mean >80% and <20%. Maybe this is British English. What you call ‘likely’
I call ‘more likely than not’.
Otherwise I agree, except 1 in 20 doesn’t seem almost impossible to me. Maybe 1 in 1000, or greater. For example: ‘it’s almost impossible you will die during this surgery’
[Greater than and less than signs converted to HTML entities. You cannot use the keyboard greater than or less than signs.–Econlib Ed.]
Matthew Moore
Aug 29 2017 at 2:38pm
EDIT: * “For me they mean >80% and <20%. What you call "Likely" I call "more likely than not"
@Effem: this was raised in a recent high-profile case in England. The judge said he was unable to give guidance. For me "reasonable doubt" is a far higher standard than that usually actually imposed de facto in the courtroom. I suspect most jurors use a "more likely than not" standard.
Q&A with judge:
Q4. Can you define what is reasonable doubt?
Answer: “The prosecution must make you feel sure beyond reasonable doubt. A reasonable doubt is a doubt that is reasonable. These are ordinary English words that the law does not allow me to help you with, beyond the written directions [he had already given them]”.
link: http://www.telegraph.co.uk/news/uknews/crime/9882943/Vicky-Pryce-jurors-given-further-directions-by-judge-in-speeding-points-case.html
David R. Henderson
Aug 29 2017 at 3:11pm
Nicely done. Compliments to Tristan.
My one edit would be to make unlikely below 45%.
Floccina
Aug 29 2017 at 3:51pm
The rule should be always put your numbers in.
John Alcorn
Aug 29 2017 at 4:16pm
@ Bryan Caplan:
Compliments to Tristan!
C. Behan McCullagh included such a glossary in his excellent book, Justifying Historical Descriptions (Cambridge U. Press, 1984), at page 52:
extremely probable = 100-95%
very probable = 95-80%
quite or fairly probable = 80-65%
more probable than not = 65-50%
hardly or scarcely probable = 50-35%
fairly improbable = 35-20%
very improbable = 20-5%
extremely improbable = 5-0%
According to Prof. McCullagh, most people would use grades roughly like these. Over the years, I have found them a helpful discipline for thinking about probabilities, and about how to express them consistently.
Your table is somewhat more fine-grained, in useful ways.
BTW, in my comment at your blogpost, Four Decades of Middle Eastern Disaster: The Proximate Cause, I considered including mathematical probabilities in my hypothetical prediction, but decided that to do so would be overkill. Tristan reminds me that I should have included them. Here is what I had in mind:
John Alcorn
Aug 29 2017 at 4:32pm
@ Matthew Moore:
Thank you for sharing the quotation and item about “reasonable doubt.”
I recall reading somewhere that “reasonable doubt” at trial is not a matter of probability. Rather, there is reasonable doubt if the prosecutor cannot provide a satisfactory answer to a relevant argument by the defense. The thought is that the prosecution must effectively neutralize every substantial point made by the defense.
Perhaps it was in L. Jonathan Cohen’s stimulating book, The Probable and the Provable (Oxford U. Press, 1977).
gwern
Aug 29 2017 at 6:47pm
The intelligence agencies, the CIA specifically, already have a standardized set of subjective probability terms: the Kesselman List of Estimative Words https://www.gwern.net/docs/statistics/2008-kesselman.pdf
(Do Tetlock & Gardner not mention that at all? It’s usually mentioned after recounting those anecdotes about people realizing how much they numerically disagree when using the same vague descriptions.)
E. Harding
Aug 29 2017 at 9:01pm
In U.S. political terminology, races are classified as toss-up, lean, likely, and safe. By my interpretation, toss-up is less than a 65% chance of a race going either way, Lean is between and 65% and 90%, Likely is between 90% and 98.5%, while Safe is 98.5% or above. Of course, others may use the terms differently from how I do.
David O'Brien
Aug 30 2017 at 2:33am
Similar to the way the Chili scale is arbitrary between two different restaurants, with food on one menu with 2 chili’s beside it is the same as another that indicates it as 3 or 5 chilis. How we discuss probability sometimes seems arbitrary & subjective.
You have devised the Scoville scale (measurement of the pungency (spicy heat) of chili peppers) of Probability.
I’d like to see this adopted by an international body such as the PMI.org
Thank you
Bedarz Iliaci
Aug 30 2017 at 2:33am
I suspect that the numerical values provide an illusory certainty and in fact, the common English terms are more precise.
What exactly one would mean by 65 percent chance of an event happening? And how is this degree of quantification made? Is there an algorithm that yields these numbers? I suppose Bayes’theorem.
But then you have the problem of justifying the quantities assigned to the priors.
Tracy W
Aug 30 2017 at 4:47am
That, if one were to give 1000 events each a 65% chance of happening, one would expect after the fact to find that about 650 of those events had gone that way and 35% not.
Eg: imagine meteorologists predict a 65% chance of rain over the course of a day. If I collected data on 1000 days where meteorologists predicted rain and found that on 648 days it rained and on 352 days it was fine all day, then it looks like meteorologists are fairly well calibrated.
Well if over 1000 cases of predictions of 65% chance of rain it turns out that it rained 648 days then we can tentatively conclude that your priors are quite good.
Steve Bacharach
Aug 30 2017 at 10:53am
Your kid sounds great, but he’s not exactly breaking new ground here:
https://github.com/zonination/perceptions
David Shera
Aug 30 2017 at 11:35am
Statistician Fred Mosteller did a paper/study on this topic. Some takeaways:
Probably is about 2/3 chance
possible is about 1/3 chance
modifier words like “very” tend to push higher.
So when President Ford talked about the Swine Flu, there was a “very real possibility”, translates to about 50%, which just like saying “I don’t know.”
Same when you hear people say “cautiously optimistic”. Translation: I don’t know.
Matt C.
Aug 30 2017 at 1:59pm
“Beyond a reasonable doubt” means “nearly 100% certainty when all doubts that are reasonable to a case such as this have been removed.”
blink
Aug 30 2017 at 11:58pm
You set a great example here, Bryan.
An interesting point is the asymmetry: “highly likely” and “highly unlikely are not quite complements, with the “unlikely” side reflecting greater certainty, and a similar case with “possible” and “probable.”
Do you see this as reflecting typical usage, a quirk of language, or your own idiosyncratic approach?
Bedarz Iliaci
Aug 31 2017 at 1:02am
Tracy W,
My question was to the probability of an event and you answered with events.
Consider the example Caplan gives:
This talks about a specific event that may come t happen–an attack on Yugoslavia in 1951. Now what could possibly be the events that are needed to make sense of the probability definition according to you?
Khanin
Aug 31 2017 at 4:59pm
Would you say that it is “almost impossible” to roll two sixes?
Patrick Haugey
Sep 1 2017 at 1:42pm
This is interesting. My brother is a detective with a fugitive squad. He was tracking a murderer. When he found him, he called his sergeant who asked him for how certain he was. He is not allowed to use 100% for liability reasons, in case of mistakes So he said 97. That wasn’t good enough to send more men. So he said 98.9. I think reason and clarity for probability such as posted above s only useful in a vacuum. The law makes certainty more variable.
Comments are closed.