Are Your Stats Lying To You?

Have you seen these folks purporting to be able to provide metrics regarding "unique" visitors to your website? Worse still, suggesting they can associate revenue, lead generation, or indeed any level of accurate behavioral analysis to these so-called unique data points? It turns out they are either ignorant, or lying. And rather than educating their consumers regarding the small reporting set they actually need in making important decisions they instead deluge them with a plethora of whiz-bang reporting to keep them lazy and entertained, without actually helping them to learn to know better quickly. Of course, with paychecks and revenue streams in jeopardy, one might be forgiven for openly wondering about the tactics of these service providers in justifying software upgrades--and increased subscription fees, using such smoke and mirror tactics.

Sadly this all seems to be characteristic of a brand of institutionalized learner that engages in corporate ladder climbing, trading the long term health of a business and the needs of the many for short term gains as they do their brief stint until the next rung becomes accessible to them. The close cousin of this for the more entrepenurial is that of building a business with the primary intent not of achieving maximum sustainability but rather that of "branding" the business as an attractive growth engine for the primary purpose of selling it at a multiplier of earnings and getting rich in the process (at the expense of the employees, suppliers, customers, and the acquiring companies shareholders, employees. etc.)

Looking at this more holistically, many of the consumers of this kind of so-called information aren't held accountable for their failings and thus large segments of the supply chain shelter severe dysfunction. Of course, maybe I'm just a jaded crazy person and this whole world-wide financial melt down is just a figment of our collective imagination.

Either way, believing that you can get accurate data regarding "unique" visitors, and it's cousin "market reach" is ridiculous. Personally, since I might be investing in your company indirectly through my mutual or index fund, I'd very much like to know when you foster this kind of activity--so I can invest more responsibly, elsewhere.


It's the Delta Stupid.

Many of the mechanisms that provide so-called unique reporting are based on browser cookies. The word "accuracy" and "browser cookies" have no business being used in the same sentence unless you are stating that there is no quantifiable correlation between the two.

It's very difficult to know how many people allow cookies to persist--and so many people just assume it's a tiny percentage of people that are smart enough to clean up after themselves.

I don't know if this has ever been true on the web. The landscape changes so quickly that during the adoption phase the data held in cookies may have been tainted for analysis purposes due to adoption issues. And, certainly over time hyper-commercialistic sites have trained us through sheer self-defense to clear our cookies with increasing frequency--if only to allow our browsers to continue working as tools for other sites we frequent. It's even a game among many to deliberately provide misinformation in the form of personas, some amusing in their blatant stupidity but others insidiously more clever in their misleading. What?!, you don't believe people are capable of this--but you do believe in the ever present threat of hackers exploiting your systems? Wow, who's paranoid now?

So, am I suggesting all statistics should just be thrown out as inaccurate? Certainly not. The data your bank provides about your account balance is a great example of a statistic you should pay very close attention to for example. And, as it turns out useful statistics exist for gauging visitor behavior on your web site as well. In fact, it's very likely those statistics are hiding in the same reports and datasets you are using to derive so-called uniques from. It turns out it's not the data at all, but rather the delta in the data that's meaningful. All actions have three possible outcomes: they either get better, get worse, or have no impact. Since two of those three outcomes have negative implications you might honestly question the value of doing anything--at least to an already functioning process, and you would be right to do so if your goal is to manage risk. However, for those folks who decide to roll the dice, you are specifically looking for an improvement in a key performance indicator. Or, in other words, you are looking for a delta indicating an increase.

This approach has implications on the development life cycle as it demands a baseline before any statistics can become meaningful, but consider the consequence: You might actually learn something from your time and effort.

It may well be that decision making based on so-called unique visitor data is no more valuable that putting it all on black and letting it ride. A better approach is to throw away the unique user data and instead monitor the delta on a stable data stream--don't bother with trying to gauge whether its unique or not. Focus instead on defining what is human traffic and what is not--that's likely to yield far more actionable results anyway.