You Get What You Measure
Thursday, October 8, 2009
I think we'd all be happier, and do better work, if we adopted an approach to measurement that was focused on creating proven value for the business.
"You get what you measure." I found this wonderful adage in this paper on measurement for agile development (PDF) (h/t Tim Malbon). And it helped me to put my finger on what I want to change about digital measurement.
The way we approach digital measurement needs to be fundamentally reconsidered. I propose that the experiences we create and the metrics we use to measure their effectiveness should be in service of creating proven valuable outcomes for the client's business.
It's a simple idea, and something that feels obvious. Yet, much of digital measurement as it's conducted currently, fails to meet that standard.
The single biggest failing of digital measurement over the past decade is that the measurements used are disconnected from results that actually have proven value for the business. Measurements are used to prove that we did what we wanted to do, but not that we necessarily did something that created value for the business. Creative agencies and their partners on the client side end up in situations where it's impossible to judge success because there's no way of proving that the objective has been achieved.
It's time to adopt a Valuable Measurement approach.
Click for full size image
In this alternative approach, the single overriding business objective is articulated by the client at the outset of the process. Then the client explicitly defines specific measurable outcomes that have been proven to be valuable in helping to achieve the business objective. These goals are then used to inform all decisions from there on out.
The creative agency designs an experience in service of achieving those valuable outcomes. And in collaboration with the client and any necessary numbers and measurement partner, a small number of key metrics are defined and agreed upon as the design comes into focus. What measurable aspects of this experience will contribute to achieving the agreed upon valuable outcomes?
And finally, temporary diagnostics can be used to adjust and optimize the experience once it's up and running. These diagnostics should be used to solve a specific challenge, and be discarded once the problem is solved.
There are a few key tenets at the heart of this approach:
What do you think? Where are the holes or problems with this approach? What would you change or add? Comments welcome.
Sources and inspiration:
Deborah Hartmann and Robin Dymond - Appropriate Agile Measurement (PDF)
Tim Malbon - Agile measurement
Sean Howard - The Myth of Social Media Monitoring
Anjali Ramachandran - Engagement vs. measurement
David J. Carr - Social Media monitoring and the spectrum of online relationships
"You get what you measure." I found this wonderful adage in this paper on measurement for agile development (PDF) (h/t Tim Malbon). And it helped me to put my finger on what I want to change about digital measurement.
The way we approach digital measurement needs to be fundamentally reconsidered. I propose that the experiences we create and the metrics we use to measure their effectiveness should be in service of creating proven valuable outcomes for the client's business.
It's a simple idea, and something that feels obvious. Yet, much of digital measurement as it's conducted currently, fails to meet that standard.
The single biggest failing of digital measurement over the past decade is that the measurements used are disconnected from results that actually have proven value for the business. Measurements are used to prove that we did what we wanted to do, but not that we necessarily did something that created value for the business. Creative agencies and their partners on the client side end up in situations where it's impossible to judge success because there's no way of proving that the objective has been achieved.
It's time to adopt a Valuable Measurement approach.
In this alternative approach, the single overriding business objective is articulated by the client at the outset of the process. Then the client explicitly defines specific measurable outcomes that have been proven to be valuable in helping to achieve the business objective. These goals are then used to inform all decisions from there on out.
The creative agency designs an experience in service of achieving those valuable outcomes. And in collaboration with the client and any necessary numbers and measurement partner, a small number of key metrics are defined and agreed upon as the design comes into focus. What measurable aspects of this experience will contribute to achieving the agreed upon valuable outcomes?
And finally, temporary diagnostics can be used to adjust and optimize the experience once it's up and running. These diagnostics should be used to solve a specific challenge, and be discarded once the problem is solved.
There are a few key tenets at the heart of this approach:
- Measurement should be in service of business goals.
- Clearly defined, measurable, valuable outcomes that can be affected through digital experiences should be used to guide decisions about what experiences to invest in.
- Key metrics should be defined during the initial design phase, in order to inform design decisions in service of creating valuable outcomes.
What do you think? Where are the holes or problems with this approach? What would you change or add? Comments welcome.
Sources and inspiration:
Deborah Hartmann and Robin Dymond - Appropriate Agile Measurement (PDF)
Tim Malbon - Agile measurement
Sean Howard - The Myth of Social Media Monitoring
Anjali Ramachandran - Engagement vs. measurement
David J. Carr - Social Media monitoring and the spectrum of online relationships
9 Comments:
Particularly like the simplicity of what you're arguing here Mike.
It's very easy to get seduced into proving something that's irrelevant.
What I take out of both this and Tim Malbon's post on Agile Measurement from last week (which you cite at the bottom of yours, if anyone's looking for it) is the need to be discerning about what's measured and why.
I am finding that because so much CAN be measured, it's frequently the case that either nothing really is (because of indecision about what to focus on first, or attempts to be comprehensive), or the wrong things are used as metrics.
Starting with a business-level set of objectives and using that as a guide throughout the entire process - including creative development of platforms and ideas - sounds so obvious but is rarely the case. A good reminder for me going forward.
Cheers, and thanks for another smart Arauz visual.
I think good branding agencies
do attempt to have this.
Can you think of some real examples or give some more tangible examples?
I think their are a lot of strategists/planners who attempt this but it fails when to translate into expression. Most creatives don't get business goals.
Excellent Mike, I wholeheartedly agree with this.
In addition we’ve found that it is not only challenging to find what to measure, but not least HOW to measure it.
As we are working with the same problem, let me share our experience.
We first tried to find what to measure. Now, one could either come up with something completely new, that the company had never seen before, or see if there already was something being measured in traditional marketing that we could copy (so not to have to get the company to adopt a whole new system for measuring success - that would be a killer :o). Adrian Ho has a brilliant post on this: http://is.gd/47Fdx
What we did was start looking at the result of post-test. The ones we looked at where value based metrics measured after campaigns finished running. But the company where used to them, and the adoption of this mindset came much easier.
Next up, we tried to figure out how to measure this stuff – as it was happening. That is the true beauty of digital – our ability to record anything. And, in comparison to the post-test, we are able to measure the experience as it is happening, not weeks or months later – when the memory has been bleached and manipulated by bad memory and prejudices.
And we can’t just blurt out and ask people stuff either, in polls inside the solution – because people don’t know what they want, and the answers would be misleading and useless.
We had to be smarter. Knowing that when people are left alone online, people would be honest and reflect their moods, anticipations, ambitions and dreams better than anywhere else – as long as they don’t know we where measuring them.
So we caught some inspiration from the car industry, and Nike ID. Where they build engaging online activities in order to get the people to play with their products in order to see what people want – without people feeling the pressure of “the poll”.
And we had to put this mindset into a product category where people really don’t personalize anything, so we had to think differently.
We are in the midst of this right now, trying to finish up the product with these ideas built into them – in order to measure the stuff you are writing about. Hopefully we will be lucky, and get what we want. We are very exited.
I know I am not concluding on anything here, but I hope the summary was interesting.
Seems simple enough, but the your step is difficult and actually where I think a lot of the difficult lies. In my experience most clients don't know the specific measurable outcomes that will lead to the objective. Seems crazy, but it's true. Particularly in FMCG where the true of objectives of sales and profit are masked by promotions on a massive scale.
Ask a client which is the single most important thing to go for and most will struggle to tell you (unless they have good econometrics). And they're tough to work out. If people get better at setting these objectives, they'll make better work, but I've not met many clients or planners that are good at it.
Nice post and I agree that a sharply focused business objective should be the destination for an exercise. The question is 'how sharp'? Very often the Business Objective tends to be the target set for the Marketing department. This was the case in point for a project my ex-agency did with an IT Services company. The 'Business Objective' was x% increase in leads from Europe (the marketing team's objective!) Apparently, there was no room for 'softer' aspects like sentiment etc. In such a scenario,how does one integrate the 'soft' with hard-nosed business objectives?
I have an even bigger question. By all this talk of measurement and RoI are we not reducing the Social Media (or digital) space into a kind of Direct Marketing exercise? Are we being overly-defensive as regards Social Media (Digital)? Just a thought....:-)
Hey Mike,
I often say that there is one clear way to know if you have a successful strategy or not. If you know what to measure and are actively doing so.
Not all digital initiatives are involved in creating an experience. This aside, I don't disagree that we need to be FAR more explicit in what we are measuring and why.
I really like this approach Helge is speaking of. To build a product/campaign around measurement that is implicit in the use and interaction with a product and/or campaign is brilliant. In digital, we truly can measure just about anything.
Knowing what to measure and, to your point, why, is the missing link.
Spot on Mike - we've got to be more focused on outcomes.
Digital - like all marketing - is simply a means to an end; it needs to deliver something before it serves any value.
From that perspective, I think your comments have relevance to other (non-digital) advertising disciplines too; the whole industry suffers from inappropriate measurement.
Understanding what we expect advertising to do, and recognising its limitations, are critical steps towards understanding how we can improve our use of each element in the marketing mix, and how we can better integrate their impact such that the value they deliver is greater than the sum of their individual contributions.
To that point, you may find this post on advertising measurement useful, as well as the comments on this subsequent perspective on why sales is a dangerous metric for measuring ad effectiveness.
But I think the real challenge stretches much further back in the marketing process: it's not just measurement that needs to be objective focused, but everything we do. More on that here.
@james
here's a practical example for a media company (whose revenue is generated from advertising).
goal: generate revenue
revenue = the number of pageviews for the site x sell-thru rate x average rate of ads sold x # of ads/page
that’s say increasing the "pageviews" variable may be a specific focus of a project.
three ways to increase pageviews:
1. get more unique visitors. (growth).
2. get people to come back more often (loyalty).
3. get people to stay longer once they’re there (engagement).
a way of increasing growth, for example, may be focusing on improving SEO.
This, as an example, can be pretty easily measured with basic web analytics (comparing unique visitors post-initiative, against your baseline).
Tracing that back up will allow you to gain visibility into the impact of your tactic on revenue. Of course, it gets more complicated when many things are going on at the same time : )
Mike, meausurement continues to obsess me. Thanks for your post, and there are some interesting comments. I am wondering how we can create a metric that makes sense. One of the biggest challenges we face is to be able to compare the kind of platforms, apps and communities that we create with other digital work and a whole raft of diverse marketing communications.
It's difficult to know how to measure a long term platform, even one that is used to deliver a series of campaigns, against a 'traditional' digital campaign. How do you measure the lifetime value of an evolving system against a campaign snapshot?
It's also difficult to know where to begin with measuring the influence of something like the the passive experience I might have from seeing a TV ad and then some billboards over a two week period with going to a brand's web platform and spending maybe 10 quite involved minutes with them a week... the numbers of the latter will be infinitesimally smaller than the numbers I can get for ads on TV and a billboard in the centre of town - but clearly the ten minutes per week, maybe every week, is clearly worth a lot more...
Quite frankly, it's doing my bleeding head in!
Time seems to
Post a Comment
<< Home