Executive Summary
When you decide to buy a product that Amazon has recommended to you, watch a movie that Netflix has suggested, or listen to a song that Spotify matched to your preferences, you are making data-driven changes to your life. Along the same lines, if you use Waze or Google Maps to decide what route to take when you are driving (or biking), check the weather forecast before you get dressed, or scan a wine bottle with Vivino before you decide what to drink, you are being data-driven. But while the average consumer of 2020 is significantly more data-driven than her counterparts were even 20 years ago, there are still many areas in which being data-driven is more of a hope or an illusion than an actual reality. One of these areas is work, as human intuition, serendipity, and bias are still stronger currencies than data. Indeed, there are few signs that the world of work has become much more evidence-based or fact-driven than it was just 20 or even 50 years ago.
Data has been hailed as the new gold, oil, and soil, out of which invaluable products, services, and progress shall arise. That may be true, but the fact of the matter is that data is just symbols, mostly numbers: zeros and ones. This applies even to the most complex algorithms, and most of what we mean when we talk about AI is classification software that assigns zeros or ones to match different variables or predict patterns at scale. To be sure, in most relevant areas of life, we still need human expertise to translate data into insights, and the willingness to act on those insights is what ultimately makes someone data-driven. Data without insights is meaningless, and insights without action are pointless.
This may sound abstract, but in many domains of life we actually operate according to these principles on a regular basis, even if we are not data scientists. For example, when you decide to buy a product that Amazon has recommended to you, watch a movie that Netflix has suggested, or listen to a song that Spotify matched to your preferences, you are making data-driven changes to your life. Along the same lines, if you use Waze or Google Maps to decide which route to take when you are driving (or biking), check the weather forecast before you get dressed, or scan a wine bottle with Vivino before you decide what to drink, you are being data-driven.
Insight Center
But while the average consumer of 2020 is significantly more data-driven than her counterparts were even 20 years ago — life was way more spontaneous, serendipitous, and error-prone back then — there are still many areas in which being data-driven is more of a hope or an illusion than an actual reality. One of these areas is work, where human intuition, serendipity, and bias are still stronger currencies than data. Indeed, there are few signs that the world of work has become much more evidence-based or fact-driven than it was just 20 or even 50 years ago.
For instance, despite the well-established fact that people enjoy their job more and feel a greater sense of purpose when their personal values and personality are a good fit with their role or job, the majority of people end up in jobs they dislike and are disengaged and unproductive at work. It is notable that these estimates come from global multinationals that employ the smartest and brightest of workers around the world, and yet these highly-educated individuals make irrational — but often predictable — career choices. If people had a better understanding of their talent and their potential, then they would be more likely to pick a job that is compatible with their interests and abilities, as opposed to something their aunt, cousin, or next-door neighbor suggested.
It is also clear that robust data — in the form of hard facts — is notoriously absent on the employer side. Consider that even the richest and biggest, most sophisticated corporations (from an HR standpoint, in the sense that they employ dozens of Ph.D. level I/O psychologists in their talent management units) are still not really data-driven in their talent management practices. As I argue in my latest book — Why Do So Many Incompetent Men Become Leaders? And How to Fix It — there is a general tendency to hire and promote on confidence rather than competence, and people who are able to charm and manipulate others in short-term interactions, such as the job interview, are far more likely to advance their careers than those who actually deliver the goods and add more value to the organization. If you think about it, all the advice devoted to making people more confident, assertive, politically astute, and better at self-branding is clearly indicative of employers’ inability to focus on substance rather than style. Alas, we live in a world where all style and no substance will get you farther than no style and all substance.
To be sure, humans are perfectly capable of rationality, and we wouldn’t be where we are if it weren’t for the fact that reason prevailed over faith, ignorance, and superstition. That said, if you go into most organizations today and ask senior HR managers to provide robust metrics to rank-order the contributions of their senior leaders or compare the most highly compensated staff members on their performance, let alone talent or potential, the answer will be far murkier and subjective than what we would get if we asked Siri whether it will rain tomorrow or not.
Interestingly, performance is hardest to measure where it matters the most. Despite a great deal of research — and almost as much theorizing — on how to conceptualize and measure job performance, data-driven approaches work well at the lower end of the spectrum. That is, when you have high-volume data, repetitive tasks, unskilled workers, and a high degree of social engineering (a legacy of Frederick Taylor’s scientific management 100 years ago), you can define and quantify individual differences in performance pretty well. The modern equivalent of this would be an Uber driver; it is nearly impossible for two evaluators to systematically disagree about the performance (and potential) of an individual Uber driver if they have access to a wealth of data comparing his or her performance to all other drivers: how many trips a day/week/month/year they completed; how much money they brought in; how many traffic accidents or fines they had; and how passengers (and I mean thousands of them) rated him/her? In contrast, if we wanted to establish whether Marissa Mayer or Donald Trump performed well in their jobs, where the stakes are clearly higher than for an Uber driver, things get more complicated. What’s the benchmark? How do we know whether other people would have done better or worse and who those other people are or should have been? Where’s the control group? Can we truly disentangle their actual actions and decisions from people’s perceptions? Related, what would happen to Tesla without Elon Musk? And how is Apple doing without Steve Jobs? (The data-driven answer is…we don’t know.)
A final and rather obvious point concerns office politics. Politics are still an influential force underpinning the dynamics of any organization. They dictate who gets hired, promoted, and fired. Importantly, those who enjoy the privilege of dominating politics — because they are the status quo — have little incentive to sanitize or sterilize the culture by making it more data-driven. When you are part of the chosen group, the establishment, or elite, why would you allow someone or something else — in this case AI or data — to reveal that you do not deserve your success and possibly even bring you down? Meritocracy is a nice idea until you realize that it’s far less meritocratic than it seems. This, however, is a painful admission to make, and it depends on having both the humility and insight to see the world through a different lens — data.
As Stephen Hawkins noted (in a quote that’s also been attributed to Daniel Boorstin, Librarian of the U.S. Congress), “The greatest enemy of knowledge is not ignorance; it is the illusion of knowledge.” If organizations — and that’s not just HR but all leaders — want to embrace evidence-based practices and data-driven decisions, there is only one logical starting point: be sufficiently humble, self-critical, and curious to understand that your instincts and intuition may be wrong and avoid being fooled by your gut feeling just because it feels good.
