Why your Klout score is meaningless

Alex Braunstein writes about Klout, a company which measures Twitter/Facebook influence:

As a Ph D statistician and search quality engineer, I [Braunstein] know a lot about how to properly measure things. In the past few months I’ve become an active Twitter user and very interested in measuring the influence of individuals. Klout provides a way to measure influence on Twitter using a score also called Klout. The range is 0 to 100. Light users score below 20, regular users around 30, and celebrities start around 75. Naturally, I was intrigued by the Klout measurement, but a careful analysis led to some serious issues with the score. . . .

Braunstein continues with some comparisons of different twitter-users and how their Klout scores don’t make much sense. I don’t really see the point of the Klout scores in the first place: I guess they’re supposed to be a quick measure to use in pricing advertising? Whatever, I don’t really care.

What did interest me was a remark on Braunstein’s blog:

Everything in life can be measured. Some quantities live on natural measurement scales: height, weight, temperature, etc. Some quantities are derived measurements: happiness, deliciousness, hunger, etc. Though all useful measurements, research has repeatedly shown derived measurements to be inconsistent and not trustworthy individually. Specifically, if two individuals tell you their happiness levels are an 8 and a 9 on a scale of 10, we have no way to know: what this means for each individual without significant amounts of context, and which individual is “happier” even if 8 is less than 9.

I think Braunstein is on to something but I would frame it slightly differently. Happiness is typically defined as responses in a survey interview. So, no, I wouldn’t want to call it a derived quantity. I’d rather call it a subjective measurement.

The problem with the Klout score is not that it’s subjective but that it’s cloudy: we don’t know what it is. To understand a cloudy measurement, one has to poke it from the outside. With Google, people have done this using google fights, google trends, etc., and they find interesting things. Braunstein tries this with Klout and finds confusion.

Which makes sense given that Klout itself seems like a tool for . . . selling itself! Sort of other notorious rating schemes such as the Places Rated Almanac and the U.S. News college ratings.

Let’s try a baseball analogy. Hits, walks, home runs, stolen bases, even goofy statistics such as “saves”–all of these are direct measurements or nearly so (accepting that we still have to handle decisions about errors, sacrifices, etc.). Batting average, on-base percentage, earned run average, etc.: these are derived quantities. Some more esoteric derived quantities such as Runs Produced will be accepted only if they offer some benefits in understanding.

P.S. More from John Scalzi. If only he’d read this blog, he would’ve known about this ten months ago!

9 thoughts on “Why your Klout score is meaningless

  1. I think Klout is a reliable and precise measure with many virtues, not the least of which is that I have a very high score on it.

  2. I'll point out that companies (including my own) use scores like Klout to find "influential" people who comment on their products. We can't reply to every tweet that says "your company sucks", so Klout helps us to figure out where to spend our resources – by responding personally to people who have a lot of "reach" or "magnification" (their terms). Klout isn't perfect but it does turn out to be useful.

    There has been a lot of research regarding proper measures of influence/authority in social networks – seminal authors in the last decade include Jon Kleinberg, Pedro Domingos and Christos Faloutsos. Twitter has some unique quirks to it: graph degree does not equate to influence due to spammers. Klout has tried hard to reduce the impact of spammers on the influence score, and they seem to have done an ok job of it.

  3. Braunstein seems to be confusing properties of measurements (like reliability) with categories of the things being measured (like subjective versus objective). Or maybe he just has never had the experience of stepping on a bathroom scale, stepping off, then stepping back on and getting a different number.

  4. maybe he just has never had the experience of stepping on a bathroom scale, stepping off, then stepping back on and getting a different number.

    Have you tried this recently? My scale reports weight to a tenth of a pound, and when you step on a second time, it always gives exactly the same number as before.

    What's more, it does this even if before stepping on the second time, you first pick up something weighing more than a tenth of a pound. I suspect that it has software inside that remembers the previous reading, and delivers exactly the same reading again as long as the new reading is fairly close to the old.

  5. Don't we have a word for this in measurement? (i.e. "validity")

    There are constructs and there are measures. All too often — and all too foolishly — we confuse the measure for the construct.

    Happiness is NOT responses to a survey. Happiness is not is studied, and attempts to measure it usually take the form of (subjective) responses to a survey. The issue is not direct vs. derived measurements, a distinction which I am quite sure is incredibly problematic.

    The issues actually is — and we need to keep this in mind — that we can't actually measure everything in life, despite what Brandstein claims. There are any number of things that we have trouble defining (e.g. integrity) or even reliably recognizing (e.g. learning), many of which we want to study. And so we instead (try to) measure something that is related or perhaps a subset of the construct we actually are interested in.

    We call this issue "validity."

    It is ridiculous to claim that we can measure everything in life and then claim that some of those measurements "inconsistent and not trustworthy," or "cloudy." We can ATTEMPT to measure everything in life. We can CLAIM to measure everything in life. But we cannot measure everything in life. Tragically, we cannot measure many of the most interesting things in life.

    And when we are thoughtful, we know that. Unfortunately, all too often we forget about validity because we are too focused on reliability.

  6. Social media is more than saying one has 10K followers or some "klout". So what? Using SM is about how to harness those 10K followers to take a required action that leads to a measurable goal. The goal can be to signup for a newsletter, attend a Webinar, download a white paper, so forth. Knowing whom the audience is, and developing content that reaches each of the audience demographic as well as psychographic make up is the key. Moreover, identifying channels that reach key decision makers is foremost. This takes research, tracking, and refinement of campaigns in order to optimize ROI. How a video will drive a Twitter feed that is created for a goal, say for attending a paid Webinar, is the synergy needed to propel desired ROI. However, there is even a larger picture. It’s how to test using A/B methods or multivariate methods, and create stronger messages from the test results at decreased costs that in turn raise ROI.
 To say that social media is not a new technology is false. However, to say that technology is evolving faster than businesses can keep pace is true.

Comments are closed.