It’s not the size of your data, it’s what you measure

Over the past few years we’ve seen an awful lot of over the top “Big Data” hype. Every use of stats wants to call itself “Big Data” and every user wants to say they’re a “Data Scientist” (what the hell is a non-data scientist, anyway?), but more importantly the uses of big data are getting oversold. The research avenues that have been opened by the relatively new ability to collect, store and analyse truly huge data sets are amazing, but every time you hear about big data replacing old fashioned things like domain specific knowledge and human judgement, you should be very suspicious. This leads to all kinds of confusion between statistical significance and genuinely meaningful information, especially in the face of a culture that systematically focusses on the wrong things.

This morning, shortly after a rather related Twitter conversation, I saw a news story that encapsulated it perfectly: a claim that Big Data was behind Yahoo!’s revocation of employees’ right to work at home. This feels like a meaningless use of the term “Big Data” in the first place given that the data in question is just VPN usage. That could have been measured when VPNs were first introduced (2000 or so, I think?), and while I hope they did some summary stats on those numbers, the first year of my Psychology BA taught all that would have been needed. But that’s hardly the main point here.

What really disturbs me is that Marissa Mayer seems to have made this decision based on a faulty measure of the wrong thing. It’s a faulty measure because relatively few workers need to be connected to their employer’s VPN every minute they’re working. It fails to capture times people print stuff out to read away from the desk, times they go offline to work undistracted, and times they meet colleagues, customers or vendors in person but outside the office. Even if it were a perfect measure of hours at work it can’t distinguish between intense focussed work and time nominally in the office but actually gossiping around the water cooler or giggling at lolcats.

But most important of all, it’s a measure of the wrong thing because it’s focussed on inputs rather than outputs. Unless all of Yahoo!’s remote workers are paid hourly, it makes no sense at all to measure hours spent at work. The only metrics that make sense for evaluating workers are output metrics: things like work done, customer satisfaction and feedback from peers (as you can probably tell, I am a fan of the Results-Only Work Environment idea). They generally feel less precise than measuring hours spent at work, but because they measure something more meaningful I would argue that they give a much more accurate picture of what management really needs to know.

This hits two big cultural blindspots that I’ve been interested in for a while, both of which are among the reasons we started the Happiness Initiative. Our culture tends to value the  appearance of precision over truly meaningful data, which helps us to be stuck with stupid metrics like hours worked, GDP, credit scores and SATs. These are all terrible proxies for what they claim to measure, but they’re also all easy to distill to a single number that looks authoritative. We also massively overvalue inputs relative to outputs and true costs: not only hours worked vs productivity but spending vs benefits from that spending (GDP!), eyeballs vs the persuasiveness of an ad campaign, or even miles of bike lanes vs the number and type of people cycling.

It gets worse! That same article goes on to suggest some genuinely Big Data metrics, like tracking employees’ movements within the office and analysing that to see how often people leave their desks, talk to each other, and so on. While this would probably add up to a somewhat more accurate picture of how and how effectively people are working, it ignores the impact of the measurement itself. Trusting employees makes them more loyal and motivated while obvious displays of distrust demotivate people. Sure, the panopticon office would help an employer select among their staff, but they would still get less out of all the good ones. And that seems to be a final cultural blind spot: an obsession with local optimisations and comparisons between people, at the expense of policies that improve the whole.

This entry was posted in Systems and tagged , , , , . Bookmark the permalink.