Chapter 5: Metrics & Success Criteria

What Gets Measured Gets Managed

Startups drown in data. Every analytics tool offers hundreds of metrics -- downloads, page views, sessions, sign-ups. The problem isn't lack of data; it's knowing which data actually matters.

The wrong metrics create a dangerous illusion of progress. The right metrics force honest confrontation with reality.

This distinction isn't academic -- it has life-or-death consequences for your startup. Consider two scenarios: In Scenario A, your team celebrates hitting 10,000 total sign-ups and writes a blog post about "10K users!" In Scenario B, your team discovers that only 3% of those 10,000 users returned after Day 7 and immediately pivots to fix the onboarding flow. Both teams had the same underlying data, but Scenario A looked at vanity metrics (cumulative sign-ups) while Scenario B looked at actionable metrics (retention rate). Scenario A wasted weeks celebrating while the business slowly died. Scenario B identified the problem and started fixing it.

The metrics you choose to track aren't just measurements -- they're decisions about what your team pays attention to. When you put "total sign-ups" on a dashboard, your team optimizes for sign-ups. When you put "Day 7 retention" on the dashboard, your team optimizes for retention. The metric shapes the behavior, which shapes the outcome. Choose wisely.

Bug #1: Celebrating Vanity Metrics

Vanity metrics are the most dangerous trap in startup analytics. They always go up over time, making the team feel good -- while the business slowly dies.

Vanity vs. Actionable Metrics

There's a subtler form of vanity metrics that catches even experienced founders: rate metrics without denominator transparency. "Our conversion rate is 15%!" sounds impressive until you learn it's based on 20 visitors. "We have 40% Day 1 retention!" sounds great until you learn it's from a cohort of 5 users. Always pair rate metrics with the sample size. Small samples produce wildly unreliable rates that can swing from 0% to 100% based on a handful of individual decisions. As a rule of thumb, you need at least 100 users in a cohort before retention rates become statistically meaningful.

The Pirate Metrics Framework (AARRR)

Dave McClure's Pirate Metrics framework gives you a structured way to think about your entire funnel. Each stage represents a different aspect of user behavior, and each has its own key metric:

The AARRR framework is powerful because it forces you to think about your business as a funnel where each stage depends on the previous one. Investing in Acquisition when your Activation rate is 5% is like pouring water into a bucket with a hole in the bottom. Fix the Activation hole first, then scale Acquisition. The framework makes these dependencies explicit.

For MVP-stage startups, the most common mistake is over-indexing on Acquisition (top of funnel) while ignoring Activation and Retention (middle of funnel). LeanPivot's Pirate Metrics (AARRR) tool helps you define and track metrics across all five stages, identifying your biggest funnel bottleneck so you focus improvement efforts where they'll have the most impact.

The North Star Metric (NSM)

Your North Star Metric is the single number that best captures the core value your product delivers. It aligns the entire organization -- product, marketing, sales, support -- around one goal.

The distinction between leading and lagging indicators is critical. Revenue is the ultimate lagging indicator -- by the time revenue drops, the underlying problem has been festering for weeks or months. A good North Star Metric predicts revenue changes before they happen. If Spotify's "Time Spent Listening" drops this month, churned subscriptions will follow next month. The NSM gives you an early warning system that lets you act before the damage is done.

North Star Examples by Category

A common mistake is choosing revenue as your North Star Metric. Revenue measures value captured, not value delivered. A company can increase revenue in the short term by raising prices or reducing service quality -- both of which destroy long-term value. A value-based North Star forces you to focus on what matters: delivering value to customers, which is the only sustainable path to revenue growth.

The Activation Metric: Your First Critical Gate

Before users can be retained, they need to be activated. Activation is the moment when a new user first experiences the core value of your product -- the "aha moment."

Identifying your activation metric requires looking at behavioral data: what do retained users do in their first session that churned users don't? Segment your users into two groups -- those who are still active after 30 days and those who aren't -- then compare their first-session behavior. The behavioral differences reveal what drives retention. Facebook famously discovered that users who added 7 friends within 10 days were dramatically more likely to become long-term users, and they restructured their entire onboarding flow to drive that specific behavior.

For MVP-stage products, you may not have enough data for this kind of statistical analysis. In that case, use your concierge and Wizard of Oz experiments to observe the activation moment qualitatively: watch users interact with your product and note the moment their expression changes from confusion to understanding. That's your aha moment. Design your onboarding to accelerate that moment.

Retention: The Only Metric That Matters

For an early-stage MVP, retention is the only metric that truly proves product-market fit. If users don't come back, you haven't built something people want.

Retention Benchmarks by Business Model

The Retention Curve

The shape of your retention curve tells you more than almost any other piece of data. A curve that flattens means you've found a group of users who find lasting value in your product -- these are your true customers. The level at which it flattens tells you how broadly your product resonates. If it flattens at 5%, you have a narrow niche. If it flattens at 40%, you have broad appeal. Both can be viable businesses, but they require very different growth strategies.

One nuance that many founders miss: retention should be measured by cohort, not in aggregate. Aggregate retention can look stable even while each individual cohort's retention is declining -- because new users mask the churn of old users. Cohort analysis separates each group of users who signed up in the same period and tracks their retention independently. This reveals whether your product is actually improving over time (later cohorts retain better) or just treading water.

Net Promoter Score (NPS)

NPS measures customer loyalty and predicts viral growth. It's simple, standardized, and benchmarkable.

NPS Benchmarks

The Sean Ellis Test: Measuring Product-Market Fit

NPS measures recommendation intent. The Sean Ellis test measures something more fundamental: dependency. If users would be "very disappointed" without your product, you've achieved product-market fit.

The 40% threshold comes from empirical analysis across hundreds of startups. Companies that crossed the 40% "very disappointed" threshold consistently achieved sustainable growth, while those below it struggled regardless of how much they invested in marketing and sales. The test is particularly useful for MVP-stage products because it requires only 40-50 survey responses to be meaningful -- far fewer than retention analysis, which requires hundreds or thousands of users.

Your Metrics Dashboard Checklist

Key Takeaways

Now that you know what to measure, let's explore how to prepare for launch and manage beta testing effectively.