Has big data changed the way many businesses operate?
Absolutely. Most e-commerce businesses today run sophisticated product recommendation engines based on big data. Amazon is reputed to generate 30% of its revenue through product recommendations. Similarly Google would be unthinkable without predictive search, autocomplete autocorrect and of course – from a revenue standpoint – its big data based ad selection. But it also extends far beyond that. Think only of self-driving cars. They are coming not just from Tesla and other startups, but from mainstream companies like Mercedes and Audi, and they would be impossible without big data.
How does human error impact the role of big data?
The promise of big data is that it somewhat lowers a chance for human error in data selection. If, as in conventional small data, one only chooses a small sample of data to extrapolate to the whole, then small errors in sample selection have a huge impact. Just ask pollsters. In contrast, when one uses close to all of the data, this problem disappears.
Or think of data only collected through one particular sensor. If that sensor has a problem, then all the data is off. If, on the other hand, in a big data context, one uses data from many different sources utilising many different sensors, then there is no single dependence on one particular way of data collection, and thus that data selection challenge again is diminished. Of course, data analysis and data interpretation need to be done well, and with full awareness of its inherent limitations (but that is not fundamentally different from small data analysis).
Humans have a tendency to seek causation and correlation when it isn’t there. Could big data lead us to false conclusions?
Yes. The temptation for humans to see causes where there are only correlations is a huge challenge, and one that we need to be aware of. The problem, however, is not big data, but what we do with the insights we gain from it, how we interpret them.
Are there other challenges for big data in business, and how can these be mitigated?
The biggest problem is to have the appropriate big data mindset. For instance, with big data, at the outset you don’t know what new insights you’ll get. So you essentially go on a fishing expedition without knowing the size of your catch. A lot of companies with traditional thinking may be wary spending money on big data analyses without having a concrete ‘use case’, a concrete way of making money, in mind. But it is precisely this leap of faith that they need to take in order to get far beyond their conventional imagination.
I have been told that instead of big data, firms should be paying closer attention to the ‘right data’. What do you make of this?
I think this is just another of the many attempts by consultants to generate business. It may sound nice, but when one actually thinks about it, it’s stupid. Because ‘right data’ presupposes that somebody knows how to differentiate between wrong and right data. But the fundamental point about big data is precisely that you don’t know at the outset what you are looking for, and need to be inspired by the pattern in the data that you see. Well, if you don’t know yet what you are looking for, how can you then make a sensible decision to select the right data?
What advice would you have for a small Southeast Asian company that wants to start using big data?
The beauty of big data is that one does not need expensive factories or even large server farms. The necessary computing power can be rented at commodity prices from cloud computing providers. What is essential is access to data and, most importantly, the right mindset. So my advice would be for smaller companies to think about all the data that they could collect but are not yet collecting. They should start doing that [as soon as possible]. And, in parallel, foster the big data mindset and see whether the data they have already yields insights.
Viktor Mayer-Schönberger is professor of internet governance and regulation at the Oxford Internet Institute, University of Oxford, and author of Big Data: A Revolution That Will Transform How We Live, Work, and Think
