Whether you’re a founder, product manager, marketer, or anyone else using SaaS metrics to make decisions for your business, I bet one of your biggest headaches is that you’re awash with data, yet simultaneously starved for actually useful insights based on that data. Am I right?

You know, Insights, with a capital ‘I’. Those golden realizations that change the way you think about your product, and which clearly translate into impactful action. We’re measuring virtually everything, but the most common question that I hear is “Ok… but is that good or bad?”. Far too often the answer is, essentially:

“Your guess is as good as mine, buddy.”

So we end up chasing false positives , burning our limited time and manpower fretting about natural ebbs and flows in our key SaaS metrics. Or, we end up celebrating nonexistent wins, and erroneously internalizing them as evidence that We’re On The Right Track. This can be fatal to a cash-strapped startup, where the margins for error are slim.

The Problem: We suck at benchmarking our SaaS Metrics

It’s impossible to know what good, bad and downright ugly is, if we don’t understand what Normal really looks like for our businesses – but we suck at it.

Think hard: what’s a baseline, normal day for your SaaS product’s key metrics? How did you arrive at those numbers? Most likely you’re mentally eyeballing it, based on a rough estimate of the past week or two’s worth of data. When put on the spot, we’re all likely to use this method. But the human brain is fallible, and we’re all doing it wrong:

The Recency and Primacy effects are well-documented psychological phenomena. They demonstrate that, when recalling a list of items (like recent metrics about your business’ performance), you’re more likely to remember the first and last numbers in the data set. We pay attention to those values more, due to their position in that set, than their actual significance. We’re neurologically hardwired to completely suck at accurately estimating what’s “Normal”. Waah, we’re doomed, right?

The Solution: Standard Deviation from the Median

It turns out, there is a way to quickly and accurately baseline your key SaaS metrics with high confidence. In this post, I’m going to show you exactly how to do it yourself, using a technique I call Standard Deviation from the Median.

“Standard Deviation from the Whatsit? Eew, is this… Math?”

Don’t fret if it sounds overly mathematical. A quick personal confession: I almost failed statistics in high school, and yet, a couple decades later, I was able to apply these methods to create Filament, a content analytics platform that was recently acquired by social data giant ShareThis. My grimly-determined statistics teacher, Mrs. Miller, would be proud (cheers, Mrs. M!). If I can handle the math theory involved here, then you’ve definitely got this—and I’m here to help!

Use the links below to dive in:

Why is Normal important to know?
What is a Normal Range?
Standard Deviation and the Median
How to calculate your SaaS business’ Normal Range
Caveats
Get your free Normal Range Calculator

NOTE I’ve spent a bunch of time researching the following material, but if you’re a real, dyed-in-the-wool statistician, and you spot ways to make this guide more accurate, please do give me a shout @king_jaffy ?

---

Ok, back up: Why is ‘Normal’ so important to know?

Because “Good” and “Bad” are squishy, subjective and relative terms. If you’re crystal clear about what’s Normal for your SaaS metrics, you’re better able to say what’s truly Abnormal. Therefore, you’re able to make better decisions as to what to prioritize.

You can more confidently tell whether that sudden spike in your metrics is actually a response to the new feature you just shipped, instead of a coincidental flutter in the metrics that’s part of the typical background noise.

Conversely, you’re less likely to derail your team’s momentum by dropping everything to investigate a sudden dip in, say, New User Signups — which often turns out to be a false alarm. Calculating your SaaS product’s Normal Range eliminates this uncertainty by telling you which values are probably normal. This gives you a solid basis with which to say “Sweet, this is a real win!” or “Dangit… Something’s broken” for any given result. You can then confidently prioritize your next steps.

It’s actually quite easy

Good news: it’s easy to calculate your key metric’s Normal Range. It will probably:

• take you about 20 minutes, (using the free calculator I’ll share with you later on in this post)
• save you and your team a bunch of time spent investigating red herrings, and possibly
• uncover some previously-hidden wins for your business, too

Sound good? Then let’s go! Don’t worry, I’ll make this the most interesting explanation of statistical SaaS metrics analysis techniques you read all week. 😉

“Listen to me, Morty. I know that new situations can be intimidating. You’re looking around, and everything’s scary and different… but *burp* y’know, meeting them head-on, charging into them like a bull — that’s how we grow as people.” ~Rick Sanchez, Rick & Morty

---

What is A Normal Range for SaaS Metrics?

Before we dive into definitions, let’s clarify our goal. We want to establish what’s Normal for our SaaS business’ metrics, so we can compare it against the peaks and valleys in our charts. We can then determine which ones are actually significant outliers, and therefore worth investigating further to figure out what caused them.

Quantifying the middle/Average/Mean of our data set is a start, but very few of the numbers in this set will land exactly on that average number. In real life, they’ll likely land some amount either above or below that line.

In addition to a precise value for the middle of our data set, we should have a buffer zone around it. For values that are within the buffer zone, we can then confidently say “Yep, that’s probably a normal result for this metric”.

This is our Normal Range, which helps us ignore results that appear good/bad to our faulty brains’ memories, but are probably quite typical. We can now focus on figuring out what’s driving those numbers outside of our SaaS metric’s Normal Range, in the weird and exciting realm of the Abnormal.

 

Standard Deviation, and the Median (an explanation for Math-haters)

We can measure two aspects of our SaaS business’ performance data for any given metric, that will help us create an accurate Normal Range: the Median, and the Standard Deviation.

The Median is the middle number in your data set, when you rank all the numbers from largest to smallest. If you have an even number of data points, it simply takes the average of the middle two numbers.

The median of 1, 2, and 3 is 2; the median of a complete set of whole numbers (integers) 1 through 10 is 5.5. Simple, right? We use the Median to determine the “central tendency” of your data set, i.e. literally, where’s the middle?

Standard Deviation (SD) is a measurement of how spread out your data is. Are your numbers all over the place, or are they pretty steady, with only minor fluctuations? If our metric oscillates wildly, the buffer zone of our Normal Range should be wider than if it constantly flutters around the middle.

An Example

If your Median website visits per day is 100, and the Standard Deviation is 50 visits, 68% of your daily traffic numbers will fall somewhere within your Normal Range between 50 and 150 visits.

Therefore, any days where you get more than 150 visits are objectively Good Traffic Days…

And any days that come in under 50 visits are pretty conclusively Bad Traffic Days (thanks, Commodus).

Classifying your daily performance this way means you can focus on picking apart these abnormal results to determine what caused them. Look for new referrers, high-performing ad campaigns, or even take it further, by testing for correlations with your product’s revenue with a Regression Analysis.

Wait, why aren’t you using the Average, like a regular flesh-human?

The Average gets really screwed up by Outliers, i.e. those fluke, extreme spikes and dips in the metrics. Take, for instance, the typical traffic spike scenario, thanks to a glowing tweet about your product from Elon Musk. What would you say is the Average number of Daily Visitors in a situation like this?

If you believe the chart above, the calculated Average performance for Daily Visitors is higher than 80% of our actual results. How can we use this to make everyday business decisions, if an Average is supposed to tell us what’s Normal?

The answer is, we can’t. The Average in this common scenario is a meaningless number, and a recipe for bad decisions that hurt your team’s morale. Who wants to see the results of their efforts fall way below average every single day, because the threshold was skewed by one ridiculously awesome day? Analytics tools do this all the time, and it drives me bonkers.

Kill it with fire — I mean, the Median

We’re using the Median instead of the Average (AKA Mean), because it’s less affected by extreme outliers in your data. The Median of a set of data containing the values 1, 2, and 100,000,000 is still 2. This makes the Median a more reliable way of measuring a realistic middle in the spiky world of SaaS metrics.

How to calculate your SaaS business’ Normal Range

Here’s what you need:

• About 20 minutes of peace & quiet
• A data sample for your SaaS metric — This could be any metric for your business, like Weekly Website Visitors, or Daily Revenue. When establishing a Normal Range for consulting clients, I try to get at least 90 data points. The more data you have, the more reliable your results.
• This handy Normal Range Calculator — For this walkthrough, I’m using Google Sheets, but no worries if you’re on team Excel, it’s all still applicable. You do you!

Get the free Normal Range Calculator

Step 1 — Import your data set

Open up a fresh new spreadsheet, and choose File » Import…, then grab your data export file.

NOTE If you’re using the free Normal Range Calculator, make sure to check the “Replace data starting at selected cell” option, to populate the correct cells instead of overwriting them.

I usually export my data sets as Comma-Separated Values (.CSV), but you don’t always have the luxury of choosing how your data is supplied to you — luckily Google Sheets is awesome, and can handle a variety of data file formats, meaning the hardest part here is usually figuring out how to export data from your analytics tools in the first place.

Step 2  —  Calculate your Median

Ok, let’s ease into this — first, rank all your data points from largest to smallest. Now, here’s the formula for calculating the Median:

{(n + 1) ÷ 2}th

“n” is the number of items in the set and “th” means the (n)th number in the set. To calculate the Median in your spreadsheet, pick a cell to contain your Median value, then input the following spreadsheet formula:

=MEDIAN(XX:YY)

XX and YY are the starting and ending cells of your data set. It should look something like this:

Boom! This is the middle number of your data set. You may also want to calculate the Average of the set, for comparison — you can either use the “=AVERAGE(XX:YY)” spreadsheet formula, or just click on the ever-smarter Google Assistant to just see it preemptively calculated for you.

Next, let’s measure our numbers in the data set, using Standard Deviation, to see how dispersed they are.

Step 3 — Calculate your Standard Deviation

Ok, take a deep breath, and behold the formula for calculating Standard Deviation:

Oops, here it is:

Yeah, I don’t see much of a difference between the two, either… Thankfully, we have no need of manually calculating such an abomination. We can let Google Sheets do it for us with its baked-in formulaic sorcery! Pick a cell you want to contain the SD value*, and enter this formula:

=STDEV(XX:YY)

* Are you sure you don’t want to use the free Normal Range Calculator, with all the formulae already taken care of for you? Just saying…

This number is the Standard amount any result in your SaaS metric’s data set is expected to Deviate from the middle, i.e. the Median.

Step 4 — Create your Normal Range

Here’s where it gets real: adding and subtracting the Standard Deviation to and from the Median gives you the upper and lower bounds of your Normal Range. You can even generate a chart to clearly visualize it, and identify the abnormal values to investigate.

Easy, right?

Now, investigate those abnormal values

To interpret your Normal Range, follow this general guideline, called The 68-95-99.7 Rule (it just rolls right off the tongue).

More info on Standard Deviation

This guideline states that in general, most unbiased data sets like ours tend to cluster their values together, with roughly 68% of the values lying within 1 Standard Deviation from the middle, 95% of the values falling within 2 SDs, and 99.7% lying within 3 SDs. These are Confidence Intervals.

In the Normal Range we calculated, the upper and lower bounds are 1 SD above and below the Median. Therefore, any results outside of that Normal Range are in the top and bottom thirds of all data points. This makes them probably worth looking into!

 

Caveats

Flukes still happen

We’re talking about probabilities here, after all. A value for your metric on any given day still has a 1-in-3 chance of falling outside your Normal Range. It’s possible that an abnormal result, hovering just outside your Normal Range, is a fluke with no directly traceable cause.

Your Normal Range shifts over time

Slow, sustained trends in your metrics don’t appear as abnormal individual values from a one-off reading. You’ll want to recalculate your Normal Range on a regular basis, so the “New Normal” doesn’t catch you off-guard!

Negative lower bounds

If your metric fluctuates significantly, the lower bound of your Normal Range will sometimes be negative. For SaaS metrics that logically can’t go below 0, (e.g. Daily Visitors), the odd 0 day is still a possible normal value.

The occasional 0 day shouldn’t be cause for major concern, but a consecutive string of them usually indicates a problem. Usually this lies either with the tracking of that metric, or your product’s performance in that area. Either way, time to go troubleshoot your key metrics to see what’s happening.

Keep Calm and Calculate Your Normal Range

As SaaS entrepreneurs, it’s critical for us to have a good idea of where Normal lies for our key metrics. Doing so makes us better at detecting when things are Abnormal — which is usually where we find the best opportunities to grow our businesses. The more sensitive your antennae are at finding abnormal performance…

…the less time you waste chasing after red herrings, giving your team more focus.

…the better decisions you can make about your priorities.

…the easier it is to know where to start investigating and looking for potential correlations with revenue.

It’s quick & easy to calculate the Normal Range for pretty much any key SaaS metric, so what’re you waiting for? Plug your data into the free Normal Range Calculator below, and start hunting those abnormal numbers. Oh, and definitely tell me what you find!

Get the free Normal Range calculator template

 

---

Strapped for time? Let me help

Learning the finer points of statistically analyzing your SaaS metrics is all well & good, but sometimes you just need it done for you, while you’re working on other stuff.

GrowthLook is a quick-turnaround, flat-fee SaaS metrics analysis service I offer, that gives you solid, actionable revenue growth recommendations tailored specifically for your business, in 7 days or less. Check it out!