In his science fiction classic the Foundation Trilogy, Isaac Asimov tells the story of the decline and fall of the Galactic Empire and its eventual replacement by a Second Galactic Empire. Inspired by the collapse of the Roman Empire, Asimov focuses the first book in the trilogy on the story of Hari Seldon, a brilliant mathematician who has developed a field of statistics that is capable of predicting the future. Called psychohistory, Seldon uses his new science to predict the fall of the Empire within 300 years and to foresee that, if no action is taken, the period between the fall of the current empire and the rise of a new one will be 30,000 years of anarchy and misery. While Seldon is too late to save the present Empire, using psychohistory he predicts that with a large enough group of followers he can chart a new course for history and shorten the gap between the old empire and the new empire to a single millennium.
Seldon is able to do all of this thanks to the power of psychohistory, which holds that while it is nearly impossible to predict the future actions of an individual, the actions of large groups can be foreseen. Or, in the words of Azimov himself,
Psychohistory dealt not with man, but man-masses. It was the science of mobs; mobs in their billions. It could forecast reactions to stimuli with something of the accuracy that a lesser science could bring to the forecast of a rebound of a billiard ball. The reaction of one man could be forecast by no known mathematics; the reaction of a billion is something else again.
Azimov explained the distinction between modeling groups as opposed to an individual by using the analogy of a gas: it is difficult to predict the behavior of an individual molecule but entirely possible to predict how a large cloud will behave. As such, psychohistory has two fundamental principles: that the population being modeled is sufficiently large, and that the population being modeled remains unaware of the results of the psychohistorical analyses. As long as these two criteria are met, Seldon is able to predict the future to highly significant degrees.
The amazing thing about psychohistory, though, is that not only does it allow for an excellent story, but it is increasingly possible today. The difference is that what Azimov called psychohistory, we call “Big Data.” Today, researchers use Big Data to find correlations between large bodies of data and seemingly infinite questions. For example, many hospitals today mine patient data to predict which ones are most likely to be re-admitted within a given time and then try to intervene in advance to prevent another costly and risky hospitalization. Or take for example a company like UPS, which places sensors on its trucks to measure the heat and vibration of key parts that are known to fail. In each of these cases, the data do not explain why exactly certain individuals need to be readmitted or why specific truck parts are failing, but the data do provide enough of a correlation for hospitals and UPS to confidently predict future results and act preemptively.
Like psychohistory in Seldon’s world, Big Data is a new and disruptive approach to measuring things. For centuries, mankind has carried out statistical modeling by using small, highly curated data sets. Because gathering information was so difficult, quality of data was substituted for quantity. Today, that relationship is being inverted. In the words of Kenneth Neil Cukier and Viktor Mayer-Schoenberger, the authors of Big Data: A Revolution That Will Transform How We Live, Work, and Think,
Using great volumes of information in this way requires three profound changes in how we approach data. The first is to collect and use a lot of data rather than settle for small amounts or samples, as statisticians have done for well over a century. The second is to shed our preference for highly curated and pristine data and instead accept messiness: in an increasing number of situations, a bit of inaccuracy can be tolerated, because the benefits of using vastly more data of variable quality outweigh the costs of using smaller amounts of very exact data. Third, in many instances, we will need to give up our quest to discover the cause of things, in return for accepting correlations. With big data, instead of trying to understand precisely why an engine breaks down or why a drug’s side effect disappears, researchers can instead collect and analyze massive quantities of information about such events and everything that is associated with them, looking for patterns that might help predict future occurrences. Big data helps answer what, not why, and often that’s good enough.
For now Big Data is incapable of predicting the future of an empire and in that regard it is nowhere near as powerful as Azimov’s mythical psychohistory. But as our ability to capture and analyze data progresses, Big Data will grow out of its infancy and become increasingly capable of finding more and more correlations that were once the realm of pure conjecture. Already today economists and political scientists are able to predict the likelihood that a given country will experience a civil war or coup. Imagine what Big Data could do for the accuracy and scope of such forecasts? It’s entirely possible that years from now a Big Data scientist will re-write Paul Kennedy’s classic The Rise and Fall of the Great Powers not as an examination of the past 500 years of geopolitics but as an accurate prediction of the future 500 years of great power rivalry. This may seem far fetched to us now, but it wasn’t that long ago that computers were considered too large and expensive to ever be practical for the average family.
The Foundation Trilogy is one of my favorite series thanks to the creativity of its author. Reading Azimov’s work is like stumbling upon the prophecies of a modern-day Nostradamus; his ability to look into the future and accurately guess what it will be like stands in sharp contrast to the predictions of his peers. Today, Big Data is increasingly providing the same function as Azimov’s psychohistory and shows amazing potential as our abilities and understanding of how to use it evolve. As is the case with many of the things Azimov wrote about, what was once solely the realm of science fiction is now reality. Perhaps somewhere in our future is a Hari Seldon waiting to save humanity from disaster. It sounds ridiculous, but so too did the idea that psychohistory would ever become a real science.