Yet another article trying to diagnose and solve a problem that every organization could likely solve differently. I’m solving this problem for companies that are in very immature stages of testing/optimization. That means you may have run one or two tests but want to run tests continuously and simultaneously. Go to any other site to find best practices. Yes, hit something like 90% statistical significance, pick 1 KPI, QA your shit, watch traffic volume, pay attention to local maximums, etc…
You’re probably doing all that. If you aren’t, click on one of those links above and read about it. It’s good stuff, but it has been said before in a better format than I have patience/talent to provide. Okay, so what is this thing that fewer people are talking about? To grow your program into a well-oiled machine you need to make tests appear less expensive. That doesn’t mean cheapen it… but let’s break down why this is critical.
Keep it simple
You’re smart. You don’t have to tell everyone else you’re smart. At the end of the day, people only want to know a few things. They want to know why we tested the thing, how we measured it, what won, and how we’ll operate as an outcome. That’s why I (alongside an incredibly talented designer) created the “3 Second Scan” template.
What’s a “3 Second Scan” template? It’s designed to be able to be glanced at and understood by ANYONE in under 3 seconds. Okay, maybe it takes a little more than 3 seconds, but we felt this was as close as we were going to get. It’s simple. It’s printable on an 8.5×11 sheet of paper. It’s lower fidelity. I can hang this on a wall. I can’t hang a PowerPoint presentation.
Okay, so is our work done here? No. We’ll still get smartypants people asking for the data… and rightfully so! That’s what an appendix for. Send out the raw data. Send out pivot tables in an Excel doc. You can make this as ugly as you want. The person who wants to dive into the data likely wants to manipulate it on their own and create follow-up hypotheses. Fantastic. Have at it! The more you use the 3 Second Scan template, the less they’ll care about drilling down to the nitty gritty details.
So what does this have to do with appearing “less expensive”? For one, it’s a lot faster. There’s less format jockeying. You’ve already defined EXACTLY what you’ll show up-front. You have a next step on the sheet – so your stakeholders are more inclined to say “Let’s move forward with your recommendation”. This minimizes overanalyzing.
The more pages in the slide deck, the more data you show, the more formatting you perform, the more ambiguous you leave the results, the more formal the ceremonies, the more you use language like a “failed test”… the more work people perceive you put into the test and the more time they think you wasted if a variant didn’t beat the control. This is BAD. It makes your stakeholders more risk-averse. People who haven’t adopted a testing mindset think we should only run tests that are “winners”. I would think the same thing if I thought my analysts were sinking hours or days into running a single test.
Change doesn’t happen overnight, but leveraging some kind of 3 Second Scan template is a good place to start. Feel free to copy this exact template. I built this in Word (I’m a masochist). I’ve used this in tests that have had 4 or 5 variants. It takes a little formatting work but we’re talking 15 minutes – not 3 hours.
You’ll still need to slice and dice the data for the folks who want to do their own analysis… and that’s OK! You probably already have raw data available and you can now just send them a spreadsheet as something like an appendix. I’ve found this process has made testing more collaborative. We received alternative hypotheses and ended up running more tests because they felt like they had a degree of control (they did – and that was a good thing).