“The Empiricism Myth” (from Paul Sztorc’s truthcoin mythbusting)

my note: Paul Sztorc is a statistician and PhD student at Yale, with whom I hope to soon be in touch.  His work on Prediction Markets (“PMs” in the article), a subject we’ll feature quite a lot here, will be absolutely critical to the future of political economy. I can’t sufficiently recommend to the reader his work, along with that of Dr. Robin Hanson, on the subject of PMs, Truthcoin, and Futarchy.
For our purposes here, Prediction Markets are a unique example of the proper empirical application in social phenomena. I add emphasis to highlight this 

Myth #2: The Empiricism Myth

The belief that certain phenomena count as evidence against the accuracy of PMs. Examples:
- “This PM was predicting that something would happen, but it didn’t happen. Therefore, PMs are inaccurate.”
 -“Source X (individual, research paper, statistical model, etc.) published a forecast that this event would occur, and did so before the PM reached that consensus. Therefore PMs are slower than Source X, which tracks the true probability more accurately.”
 -“I do not believe PMs are accurate because Source Y investigated PMs and concluded that they…” Pre-outcome, the claim that one forecast method is better than another is epistemologically impossible. Phrases such as ‘true probability’ and ‘most accurate forecast’ are laughable. Post-outcome, such claims are possible, but likely more difficult than the layperson may suspect.

For a start, the front-runners (probability > 50%) should not always win. In fact, if they did, that would indicate that they were consistently underpriced, and be evidence against the accuracy of PMs. PMs are unique to the forecasting world in that their operation and methodology are completely transparent and reproducible. Moreover, only PMs provide a publically available forecast at each moment of their existence, in contrast to a poll or research paper whose results are published one time on a single date. As such, PMs are immune to publication bias, as they can neither censor nor cherry pick their methodology or results. This immunity is highly significant, as publication bias causes roughly 60%2 to 90%3 (or more4 ) of the university-grade research-findings claimed to be true to actually be false.

For individual bloggers, TV pundits, or journalists, or other info-prostitutes, who lack scientific training and the controls imposed by peer review, the effects of selective-publication can only be even more detrimental. PMs are not accurate because they have a track record of accuracy. They are accurate because of qualities inherent to their definition as an incentive-compatible meta-tool. The accuracy is not a mysterious result, which “for some reason” we continue to observe empirically, and ultimately generalize by way of induction. The accuracy of a PM is produced by way of information-aggregation, in a completely clear and atomically understood process. If PMs are ever to be discredited, it can only be on the grounds that they fail to efficiently integrate some existing knowledge.

2 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124

3 http://www.jove.com/blog/2012/05/03/studies-show-only-10-of-published-science-articles-are-reproduciblewhat-is-happening

4 http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-itnot-trouble


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s