Harvard Business Blogs -
In previous posts, we discussed the common challenges that entrepreneurs face — from those in the garage to innovators inside large companies. In order to tell if we're making progress, we turned our attention from the outputs of models — from things like gross revenues and profits — to the inputs, insisting that the path to successful innovation is to prove the viability of the concept in micro-scale.
The idea is simple. Establish baseline metrics by building the minimum viable product — the minimum required to measure the response of early adopters. Then, in each development cycle, use the insights gained by studying customers to make improvements. This is the source of validated learning — proof that the customer insights translate into tangible metrics improvements.
But this leaves a very difficult problem still to be solved: How do we know that these changes are what actually effect change in the metrics that we're observing?
This is the curse of vanity metrics, numbers which look good on paper but aren't action oriented: website hits, message volume, or "billions and billions served." They look great in a press release, but what do they accomplish?
Consider a scenario where a team makes a product change, and the very next month page views go up. As humans, we're hard-wired to infer causality from correlation: when the numbers go up, we tend to take credit. But when the numbers go down, we tend to blame someone or something else. Worse yet, different team members tend to attribute positive changes to whatever project they were working on at the time (but not negative changes, of course). As a result, different parts of the team are constantly "learning" in their own private reality. When those teams face difficult choices, it's incredibly hard for them to come together and make an informed, fact-based decision.
To avoid falling into this trap, I recommend you follow the three A's of metrics. All metrics should be actionable, accessible, and auditable.
Actionable: When an employee sees a report about a specific metric, it's essential that they have some idea how to replicate the result in the report. Techniques like split-testing, where discrete groups of customers see different versions of the product, are the gold-standard here. Following the scientific method gives confidence that the observed behavior was, in fact, caused by the change being tested.
Accessible: Most data warehousing systems provide reports that are too complicated to read and take too long to generate. As a result, many teams don't get any benefit out of them. In order for reports to be worthwhile, it's essential that:
- Everyone in the company understand how to read them
- Everyone in the company has easy access to the latest data
Additionally, to keep accessibility high, the feedback loop between taking an action and seeing the results should be as short as possible. In my experience, waiting for even a weekly round-up or having to ask a data warehousing expert to run a report creates too much friction. Key data should be available to any employee, anytime, in a matter of minutes. In order to achieve that goal, the reports themselves have to be extremely simple.
Auditable: The biggest benefit of actionable metrics is that they can be used to reap the biggest savings in all of product development, when they tell you that you don't need to do something. Metrics have to be credible to the people who drive the product vision, including company founders. Even if they understand a report, and know what caused it, that doesn't mean they will actually learn. Is the report accurate? Believe me, when it's your idea on the line, it's much easier to believe the report is the problem rather than the idea.
Thus, it's important that skeptics can audit a report. When possible, this should mean that report generation is simple. Hopefully it can be created with direct access to primary data. For example, a report about revenue that's generated directly from the master orders database is more credible than one that requires several intermediate steps. Most importantly, it should be possible to translate the summary numbers in the report back to the actual customers who generated them.
Remember, metrics are really reports on people. This is where most off-the-shelf metrics packages fail. For example, consider a report that claims in a split-test between two proposed features, feature A generated more revenue than feature B. Which people used feature A? Which people used feature B? Can a skeptic call them up and ask them questions about their use of each feature? If not, how can we generate actionable insights about what to test next? Or consider a report that reports abstract quantities, like website hits or attrition rates. These are hard to visualize. But if instead our metrics say a sports stadium full of people declined to buy our product. Ouch. We can all viscerally relate to that.
I have found that most entrepreneurs — in any sized company — will readily agree that better analytics leads to better results. But how do they know when it makes sense to make infrastructure investments?
In order to make sense of these trade-offs, we need a heuristic for evaluating the value of new processes, tools, or infrastructure in an entrepreneurial context. That will be the subject of the next post in this series.
Eric Ries is the author of StartupLessonsLearned.com and is an adviser to many startups, companies, and venture capital firms.
It looks like a scene from another Dragnet remake. But the streets of San Francisco are being tagged with an elaborate marketing campaign for BioShock 2, a new video game from 2K Games.
Tens of thousands of Chinese who came to Russia to trade are rethinking their futures after the closure of the massive Moscow market where they traded their wares.
New research shows that dying of a broken heart isn't just a metaphor. WSJ's Ron Winslow talks with Simon Constable about studies that show real, and sometimes fatal, changes can occur in the heart after a traumatic breakup or death of a loved one.
Google is set to unveil a new feature of Gmail designed to make it easier and faster for users to share media and status updates with friends. WSJ's Jessica Vascellaro joins the News Hub from San Francisco to discuss.
In the hopes to continue a much-needed conversation on how to revamp the economy, this week I would like to highlight another policy recommendation that emerged from the Kauffman Foundation’s 
