Klaviyo A/B testing: what to test, what to ignore and how to read the result
Most Klaviyo A/B tests are wasted, not because testing is a bad idea, but because brands test the wrong things, on samples too small to mean anything, and then call a winner from a difference that is really just noise. Done properly, A/B testing is how you compound small wins into a materially better account. Done carelessly, it is theatre. This guide covers what to test in order of impact, how to size a test so the result is real, and the things you should never bother testing at all.
Why most A/B tests tell you nothing
There are three failure modes, and nearly every wasted test is one of them.
- Testing trivia. Two button colours, a comma in the subject line, an emoji. The effect, if any, is so small you would need a list of millions to detect it.
- Calling it too early. Variant A is 8 percent ahead after two hours, so you declare it the winner. With 40 conversions that lead is well inside the margin of noise and will often reverse.
- Testing two things at once. New subject line and new hero image together. If it wins, you have no idea which change did the work, so you have learned nothing you can reuse.
A good test changes one variable, on enough volume, judged on a metric that pays you. Everything below serves those three rules.
The Klaviyo tools you actually have
Klaviyo gives you two distinct testing surfaces, and they behave very differently.
Campaign A/B tests
On a one-off campaign you can split your audience between variants: different subject lines, content, from-names or send times. Klaviyo can send each variant to a portion of the list, wait, then send the winner to the remainder based on a metric you choose. On a large list this resolves quickly because all the volume hits at once.
Flow A/B tests
Inside a flow you can add an A/B test that splits people between message variants automatically as they move through the automation. This is powerful because a flow runs forever, so the test keeps gathering data, but it is slow: volume trickles in profile by profile, so a flow test can take weeks to reach a trustworthy result. That patience is the price of testing your always-on revenue, and it is one more reason we lean on flows rather than campaigns for the revenue that should be automated.
What to test, in order of impact
Test the big levers first. If you only ever run a handful of tests, spend them here.
Subject line and preview text
The highest-frequency, highest-leverage campaign test. The subject line decides whether the email is opened at all, and small angle changes (curiosity versus clarity, benefit versus product, question versus statement) can move engagement meaningfully. Test the angle, not the punctuation.
Send time and day
When your audience is most likely to buy is genuinely brand-specific, and worth testing rather than copying a generic "best time to send" chart. Test a clearly different time or day, not 9am against 9:15am. Klaviyo's Smart Send Time feature can also help you learn this per profile once you have volume.
The offer
The single biggest lever on revenue. Percentage off versus money off, free shipping versus a discount, a gift with purchase versus a straight code. Because the offer changes behaviour so much, offer tests reach significance faster and teach you the most about your customers.
Creative and layout
Image-led versus text-led, long versus short, one hero product versus a grid. These matter, but the effect is usually smaller than subject line or offer, so they need more volume to call. Test a genuinely different concept, not a nudged margin.
Flow timing and delays
Inside automations, the delay between emails is a real variable. Does the abandoned cart reminder do better at one hour or three? Does the second welcome email land better at 24 hours or 48? Timing tests inside flows are slow but valuable, because you are tuning revenue that runs on autopilot for years.
Sample size and significance, without the jargon
This is the part that separates a real test from wishful thinking. You do not need a statistics degree, but you do need to respect three ideas.
- Judge on conversions, not opens. A subject line can win on opens and lose on orders. Wherever the volume allows, decide the winner on placed-order rate or revenue per recipient.
- Small differences need big samples. Detecting a 2 percent lift reliably takes far more data than detecting a 20 percent one. If your list or segment is small, only test changes big enough to show up.
- Aim for roughly 95 percent confidence. That is the conventional bar for saying a result is unlikely to be chance. In practice it means gathering dozens of conversions per variant at a minimum before you trust a placed-order result, not dozens of opens.
The honest consequence: if you send to a few hundred people and get a handful of orders, you almost certainly cannot call a winner, and pretending otherwise just adds superstition to your account. This is also why brand-new or small lists should focus on list growth and the basics first, the groundwork we cover in our DTC email marketing strategy, before pouring energy into split tests.
What not to test
Saying no to a test is a skill. These rarely earn their place:
- Button colour and micro-copy. The classic "we tested the button" story is mostly folklore for email lists your size. The effect is too small to detect and too small to matter.
- Low-traffic segments. A test on a segment that produces two orders a week will never reach significance. Optimise it by judgement, not by split test.
- Multiple variables at once. Tempting when you are impatient, but it destroys your ability to learn. If you must, that is multivariate testing, and it needs even more volume.
- Everything, forever. Testing has a cost in time and focus. Once a variable has been settled, ship it and move to the next lever rather than re-litigating it every send.
There is also a deliverability trap hiding in aggressive subject-line testing: chasing opens with clickbait can lift opens while quietly training spam filters and eroding trust. Keep the deliverability fundamentals in view, because a "winning" subject line that hurts inbox placement is a loss dressed as a win.
A simple testing cadence that compounds
You do not need a test running every hour. You need a steady, honest rhythm.
- Pick one lever a month per surface. One campaign variable, one flow variable, rather than a scattergun.
- Write the hypothesis first. "I think a benefit-led subject line will beat a product-led one because our audience buys on outcome." Now the result teaches you something, win or lose.
- Record what you learn. A shared log of what won, what lost and what was inconclusive stops you re-running the same test in six months.
- Re-test occasionally. Audiences and seasons shift, so a settled result is worth revisiting once a year, not weekly.
Tie the wins back to the numbers that matter. If you are unsure what "good" looks like before and after a test, our guide to Klaviyo benchmarks gives realistic ranges to measure against, and the revenue calculator helps translate a lift into pounds.
Why this is specialist work, and where Nelvio comes in
Read back over it: choosing the right lever, sizing the test, resisting the trivial ones, waiting for significance instead of your gut, and judging on revenue rather than the vanity of opens. None of it is difficult in isolation. Doing it consistently, correctly and without fooling yourself, while running the rest of the account, is the hard part, and it is where a lot of "we tested it" folklore quietly costs brands money.
This is the work we do for the brands we run Klaviyo for. For Eternal Collagen we rebuilt the core flows and generated an extra £90,247 in email revenue in four months, growing the list from around 500 to over 11,000 across six live flows, with disciplined testing feeding the improvements rather than guesswork. We are not claiming that is typical or guaranteed, only that it is what proper, senior-led optimisation looks like.
If you would rather have someone run the tests that actually matter than spend a year calling false winners, start with a paid Klaviyo audit and we will show you exactly which levers in your account are worth testing first.
Frequently asked questions
How long should a Klaviyo A/B test run?
Long enough to gather enough conversions per variant to reach roughly 95 percent confidence, not a fixed number of days. For campaigns that often means a few hours to a day on a large list; for flows it can take weeks because the volume trickles in. The rule is to size the test by conversions, not by the calendar.
How big a sample size do I need for an email A/B test?
It depends on the effect you are trying to detect. Small differences need large samples. As a practical guide, you want dozens of conversions per variant at minimum before a placed-order result means anything, which is why low-traffic segments and small lists are hard to test reliably.
Can you A/B test flows in Klaviyo, not just campaigns?
Yes. Klaviyo has an A/B test option inside flows that splits people between message variants automatically. It is powerful because flows run forever, but because volume accumulates slowly you need patience and enough traffic before the result is trustworthy.
What should I A/B test first?
Start with the variables that move revenue most: the subject line on campaigns, and the offer or the first email in a flow. Test the big levers before you ever touch small details like a button colour, which rarely changes the outcome enough to detect.
Why did my A/B test show no clear winner?
Usually because the sample was too small, the difference between variants was trivial, or you tested more than one thing at once. A no-result is still useful information: it often means the variable you tested does not matter as much as you thought, so move on to a bigger lever.
Should I test for open rate or revenue?
Judge on revenue or placed-order rate wherever you can. Open rate is unreliable since Apple Mail Privacy Protection inflates it, and a subject line that wins on opens can still lose on sales. Optimise for the metric that pays you.
Stop calling false winners
We run the tests that move revenue and ignore the ones that do not, then bank the wins into flows and campaigns that keep earning. Start with a £499 Klaviyo audit and we will show you exactly which levers in your account are worth testing first, and which you should leave alone.
Book a £499 audit →