The Economist: Data Overload-The Digital Universe is Growing Too Fast
March 8, 2010
The February 25 edition of The Economist included an insightful article, “Data, Data, Everywhere,” about the digital information overload. It discusses the fact that the digital universe is growing faster than the capability to store this information. In the graph below you can see a representation of this phenomenon measured in Exabytes. (5 Exabytes = all of the words ever spoken by mankind.)
The business of information management-helping organisations to make sense of their proliferating data is growing by leaps and bounds. In recent years Oracle, IBM, Microsoft and SAP between them have spent more than $15 billion on buying software firms specialising in data management and analytics. This industry is estimated to be worth more than $100 billion and growing at almost 10% a year, roughly twice as fast as the software business as a whole.
- Wal-Mart, a retail giant, handles more than 1m customer transactions every hour, feeding databases estimated at more than 2.5 petabytes-the equivalent of 167 times the books in America’s Library of Congress
- Facebook, a social-networking website, is home to 40 billion photos.
- Decoding the human genome involves analyzing 3 billion base pairs-which took ten years the first time it was done, in 2003, but can now be achieved in one week.
- The amount of digital information increases tenfold every five years. Moore’s law, which the computer industry now takes for granted, says that the processing power and storage capacity of computer chips double or their prices halve roughly every 18 months.
Joe Hellerstein, a computer scientist at the University of California in Berkeley, calls it “the industrial revolution of data”. Hal Varian, Google’s chief economist, predicts that the job of statistician will become the “sexiest” around. Data, he explains, are widely available; what is scarce is the ability to extract wisdom from them. If you would like to learn more about this topic, listen to our free, 10 minute video presentation entitled, “Growing your Data While Shriking Your Data Center.”