Big Data is never going to beat execs trusting their gut

Image for post
Image for post

The Guesswork Era isn’t a real concept — I just invented it right now — so let me try and explain first what it means. In the simplest terms, it’s an executive (“senior decision-maker”) over-relying on their gut instinct when there’s a bunch of information (“big data”) they could base a decision on. Here’s an example:

HR Manager: “Well, sir, we have lots of information on what would make someone successful in this role, and as a result, Joseph is the bes — “

Executive: “No time, Susan! Joseph didn’t impress! Sam! I like the cut of Sam’s jib!”

The situation above could have been “People Analytics,” but instead it became the Guesswork Era. See what I mean?

This happens a lot. It’s not too complicated why it happens either. Most executives don’t understand data that well, and the way it’s often presented to them doesn’t jive with how other things (i.e. balance sheets) are presented to them. For this reason, you could argue two things: (1) is that Big Data might make decision-making slower (odd) and (2) is that for this all to work, we need “data translators” as a job of the future.

Executives want to make decisions off their gut and relevance. Big Data may not change that.

Everything I’ve said so far is about the logistics/process of the supposed Big Data era. I haven’t even touched on the psychology. Work is a very psychological place, although we often forget this and try to drown everything in “logical” processes. (Most processes are just invented to make a middle manager feel as if they’ve controlled a situation properly.) Stephen Dubner, who is smarter and more famous than I am, has also noted that a major problem with “data-driven decisions” is that executives want to believe in their gut. In the mind of some of these guys, they arrived at their perch for a specific set of reasons. If “Big Data” removes some of these reasons, how relevant are they anymore? (Not much.) Work is largely a quest for relevance towards self-worth, so who wants 750 rows of information if guesswork makes you feel better?

This is a topic we need to consider more.

The Guesswork Era and the science of selling

Here’s an article from Wharton on “The Science of Good Salesmanship.” I’d classify this as mostly interesting, although at some points the interviewee is like “Data will save us all!” Down near the end, there’s this:

True. Or, wait. We want this to be true. It isn’t actually true. The scientific data should take the guesswork out of how we make decisions, but it doesn’t. We’ve created 90 percent of the world’s data in the past 5–10 years, right? At the same time, we have some of the highest levels of decision-making variability in executive history. Shouldn’t all this data be putting our senior leadership teams on the same page, as the data is saying the same stuff to them? Logically, this would happen. But in reality, it doesn’t happen.

Why not?

Some reasons are described above. Other reasons:

  • The Silo Effect is pervasive in business, and many executives contextualize decisions relative to their silo — not to the overall company
  • A deep belief in data signals to some people that the robots are coming too
  • Company-building and market share-stealing is the closest thing some guys get to fun, so damned if an algorithm is going to take that away
  • A lot of “big data plays” are really just one company copying another company, and not one company having any kind of strategy around data
  • The people companies have in place as their “data team” usually are a bunch of target-whiffers who maybe understand data but don’t understand how to explain that data to anyone else

This all brought us to the Guesswork Era. We want to think this will change with more data, but it probably won’t.

A quick little story on the Guesswork Era concept

Had a job a while back where I sent Analytics reports every Friday. Website, apps, etc. Performance stuff. There’s a concept called “the tyranny of old metrics,” meaning that business changes but people cling to old numbers that used to matter. That idea was big-time in play on these emails. Most people had no clue what I was writing about, so I put some jokes and pop culture references in the emails. Eventually that rubbed a few people the wrong way, my boss snarled at me, and a few months later, I was shit-canned out the door two weeks before Thanksgiving. Hierarchy, baby!

So when I’m dead and buried at this joint, another kid takes over the Friday analytics emails. The first one he sends, he’s got about 700 rows of data in it. Mine were just images and text with some numbers sprinkled in. What happens? People go nuts. They’re forwarding his boss. “What is this? I am swamped! No time to read this!” Of course, most people could have deleted it — but people love to bitch, and bitch they did.

This illustrates (to me) a key point about the implementation of big data. It’s not about collecting the data or having it to show. That’s what most companies think. It’s about (a) empathy for how other people want to interact with it and (b) using it for better decisions. That’s it. Most companies completely miss that, and as a result, the Guesswork Era is likely to live on for another decade or more.

Your take on guesswork vs. data?

Written by

Blogging, largely about work and how to improve it. How I make (some) money:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store