What is A/B testing?

A/B test­ing is a method used to com­pare two or more vari­ables (vari­ant A and vari­ant B) to deter­mine which one deliv­ers more desir­able results.

Also known as split test­ing, A/B test­ing is used in dig­i­tal mar­ket­ing to mea­sure the effec­tive­ness of a range of vari­ables across web­sites, apps, email cam­paigns, and adver­tis­ing. Mar­keters will con­duct split test­ing to deter­mine which fac­tors encour­age more con­ver­sions, engage­ment, sales, shares, etc.

In e‑commerce web­sites, suc­cess­ful A/B test­ing can cause a 50% increase in the aver­age rev­enue per unique vis­i­tor. With these poten­tial­ly incred­i­ble results, per­haps it’s time your mar­ket­ing team began split test­ing – but is it applic­a­ble to your business?

A/B test­ing can enlight­en teams on a range of goals, as this sim­ple process can be applied to all sorts of sce­nar­ios. For exam­ple, A/B tests are used by shops to deter­mine the most effec­tive price points, by politi­cians to bet­ter under­stand vot­ers, and by web devel­op­ers deploy­ing new ver­sions of appli­ca­tions – just to name a few.

With­in the realm of dig­i­tal mar­ket­ing, how­ev­er, split test­ing main­ly focus­es on:

• Web­site optimisation
• Improv­ing email mar­ket­ing campaigns
• Opti­mis­ing adver­tis­ing meth­ods (e.g., pop-ups or Google Ads)

With 77% of organ­i­sa­tions run­ning tests on sites, cor­po­rate web­sites are “the most com­mon tar­get of A/B test­ing.” Land­ing pages are the sec­ond most com­mon, with 60% of organ­i­sa­tions test­ing them with the split method.

More specif­i­cal­ly, mar­keters use A/B tests to inves­ti­gate the effec­tive­ness of things like:

• Head­lines and sub­ject lines
• Copy
• Images and graphics
• Colour schemes
• Pop-ups
• CTAs
• Dis­counts and spe­cial offers

You might com­pare the loca­tion of these vari­ables, e.g., whether a CTA is placed at the end or mid­dle of an email. With copy, you could com­pare the length of the text or its tone. Per­haps you’ll want to com­pare the tim­ings of when emails are sent or com­pare how dif­fer­ent audi­ence seg­ments engage with the same email design. Ulti­mate­ly, there are end­less ways you can test dif­fer­ent features.

Whichev­er ones you choose, test­ing vari­ables in mar­ket­ing pro­vides busi­ness­es with raw data, allow­ing them to make well-informed deci­sions with confidence.

First, though, you’ll have to learn how to do it effec­tive­ly – and that’s what this arti­cle will explain. Plus, there’s the unfor­tu­nate fact that A/B test­ing mar­ket­ing isn’t the answer for everything.

This arti­cle will explain the ways in which A/B test­ing can help your busi­ness­es – as well as the ways in which it can’t help.

And, final­ly, though there are some incred­i­ble rewards to gain, does your spe­cif­ic busi­ness real­ly need it? Let’s dive in.

Why do businesses use A/B testing? The benefits:

If you exe­cute your test cor­rect­ly and plan for all even­tu­al­i­ties, you can expect to see some amaz­ing rewards.

Armed with the most accu­rate pic­ture of what dri­ves leads to con­vert, you’ll see increased sales, con­ver­sions, and engage­ment, as well as reduced bounce rates and aban­doned carts.

So, if you weren’t con­vinced of the ben­e­fits of A/B test­ing, here’s what’s wait­ing for you:

Increase in conversions

A/B test­ing can help increase both con­ver­sions and sales vol­ume. Since test­ing can improve user expe­ri­ence, opti­mise “click­a­bil­i­ty,” and refine lead nur­tur­ing process­es, boosts in con­ver­sions and sales are inevitable.

Some­thing like an improved user expe­ri­ence can pro­vide a domi­no effect. An opti­mal expe­ri­ence means users have greater trust for your brand, high brand affin­i­ty, and keep com­ing back for more.

Reduced bounce rates

If you’re con­cerned about those areas of your web­site that have high drop-off rates or low con­ver­sion rates, you can iden­ti­fy improve­ments with A/B testing.

Mea­sur­ing vari­ables such as head­lines, copy, design, and colour schemes can steer you in the right direc­tion towards reduc­ing bounce rates and keep­ing site vis­i­tors engaged for longer.

Increased user engagement

You can use A/B test­ing to improve engage­ment rates because its insights can show you what aspects of your con­tent pos­i­tive­ly influ­ence user engagement.

If you test­ed the colour of your CTA but­ton, you might observe that red saw more clicks than green. A vari­able as small as this can have a huge impact.

Less abandoned carts

For those busi­ness own­ers with­in e‑commerce, aban­doned carts are one of those illu­sive pains – often recur­ring and unex­plained. Split test­ing can help iden­ti­fy the required changes that will push site vis­i­tors over the fin­ish line.

Improved content

When test­ing sales, ad, site, and email copy, the process involves sift­ing through inef­fec­tive lan­guage to ulti­mate­ly pro­duce the best copy pos­si­ble. Writ­ers and mar­keters can learn a lot from this process and become pro­fi­cient in writ­ing per­sua­sive copy that engages and inter­ests vis­i­tors – even beyond the test­ing period.

Less risk

When we talk about risk in dig­i­tal mar­ket­ing, we’re talk­ing about the risk asso­ci­at­ed with wast­ing time, mon­ey, and resources on strate­gies that aren’t going to give you a return on your investment.

By con­duct­ing A/B test­ing on new site, ad, or email fea­tures, you can ensure your time, mon­ey, and resources are spent cau­tious­ly and con­fi­dent­ly. With data sup­port­ing your deci­sions, you can make changes to your mar­ket­ing strat­e­gy with less risk than if you were mere­ly fol­low­ing your “gut.”

More straightforward decision making

Split test­ing trans­forms deci­sion-mak­ing process­es. With raw data back­ing up cre­ative ideas, your next steps could­n’t be clearer.

What are the risks associated with A/B testing? The negatives:

Although 63% of organ­i­sa­tions find A/B test­ing easy to imple­ment, 7% said it’s a daunt­ing process. Well, what about the oth­er 30%?

This group don’t find it daunt­ing or easy – but they do have some issues with con­duct­ing A/B testing.

There are some prob­lems that can arise with run­ning tests, but you can pre­pare for them. Here are the issues you can expect to face and how to cope:

It can only help with specific goals

A/B test­ing can help with: mea­sur­able KPIs such as clicks, bounces, shares, and aban­doned carts.

A/B test­ing can’t help with: vague fac­tors like web­site ease of use or vis­i­tor frustration.

These issues aren’t mea­sur­able, and test­ing some­thing like bounce rate won’t explain why users are leav­ing. If it’s because your site is bug­ging, this is some­thing you’ll have to fig­ure out with­out the help of split tests.

It can use up time and resources

Com­pared with oth­er forms of test­ing, split tests can take a while to set up. In some com­pa­nies, there are end­less long meet­ings to dis­cuss the tests and agree on variables.

Even once these meet­ings are over, it’s time for the coders and design­ers to get to work – dou­bling their usu­al work­load in pur­suit of mak­ing two variants.

Once the tests are pre­pared, you must wait for the test­ing peri­od to pass, which can be any­where between 2 weeks to sev­er­al months, depend­ing on site and mail­ing list size.

You can work around this issue by only con­duct­ing split tests if you can actu­al­ly spare the time and resources. Though frus­trat­ing, you must­n’t short­en the test peri­od unnec­es­sar­i­ly as this will dam­age the val­ue of your results.

It won’t help all your problems

A/B test­ing can only take you so far. If your web­site or email cam­paign has core usabil­i­ty issues, no amount of tweak­ing images and sub­ject lines will help. Fur­ther­more, split test­ing isn’t like­ly to reveal these issues.

Though you might see vari­ant A per­form­ing bet­ter than vari­ant B, fix­ing core usabil­i­ty flaws (if they’re present) will sky­rock­et results much quicker.

Before you con­duct split test­ing, ask a web devel­op­er to check your site for func­tion­al­i­ty issues. If you fix any present issues, wait a month to see if things improve. Once no usabil­i­ty issues are present, you can go ahead with a split test, know­ing that the results will be based pure­ly on the vari­ables you’re testing.

When shouldn’t you use A/B testing?

A/B test­ing def­i­nite­ly isn’t the solu­tion for every­thing. Here are some instances in which you would­n’t con­duct a split test:

If your sample size is too small

Sam­pling errors can occur when too-small groups are test­ed. To guar­an­tee mean­ing­ful results, your sam­ple size must be sta­tis­ti­cal­ly sig­nif­i­cant. This is much eas­i­er for well-estab­lished com­pa­nies, whilst start-ups may strug­gle if they have small­er mail­ing lists.

To fig­ure out if your sam­ple size will be sta­tis­ti­cal­ly sig­nif­i­cant, you can use this free cal­cu­la­tor.

If you don’t have enough time to dedicate to managing it

Run­ning A/B tests is inten­sive on both time and resources. Not only are there mul­ti­ple team mem­bers need­ed to set them up, but time must be spent analysing and imple­ment­ing the data after­wards, too.

Although A/B tests can be straight­for­ward, they can quick­ly use up a busi­ness’s ener­gy if they’re overcomplicated.

As much as you might like to con­duct a split test, make sure you can spare the fund­ing, time, and resources. If you can’t, your test is like­ly to have holes, caus­ing the results to be less valuable.

If taking action is low risk

If you’ve got a low-risk idea that’s like­ly to have a pos­i­tive effect on your emails, adverts, or web­site, there’s no rea­son why you’d spend time and mon­ey test­ing it.

If time or resources are par­tic­u­lar­ly scarce, it’s impor­tant that you don’t waste them on test­ing ideas that will almost cer­tain­ly have a pos­i­tive outcome.

How to conduct A/B testing

1. Create your hypothesis

Do you think that short­en­ing your email sub­ject lines will cause more recip­i­ents to read and engage with your emails?

Or do you think more site vis­i­tors will sign up to your mail­ing list if the pop-up is placed in the mid­dle of the window?

Make sure your hypoth­e­sis is clear, sim­ple, and focus­es on one sin­gle variable.

2. Identify KPIs and goals

These are the met­rics you’ll use to deter­mine which vari­a­tion per­forms best. You can choose things like prod­uct pur­chas­es, mail­ing list sign-ups, clicks, or shares.

3. Create your variations

Your web design­er or devel­op­er will cre­ate two ver­sions of what­ev­er it is you’re test­ing (e.g., an email, advert, or mail­ing list pop-up). One is the “con­trol”, and the sec­ond is the “chal­lenger.” Your con­trol must be the ver­sion that exists already.

You can use an A/B test­ing soft­ware for this (such as Google Opti­mize 360, AB Tasty, VWO, Adobe Tar­get, Opti­mize­ly, or Ora­cle Maxymiser), but it’s not entire­ly necessary.

4. Set your sample groups

Your sam­ple groups must be equal and select­ed randomly.

5. Set a length of time to run the test.

To ensure you have a large enough data set, your test must run for a suf­fi­cient length of time. Equal­ly, if it runs for too long, you run the risk of bias.

Two weeks is the advised peri­od to run a test, as this allows you to account for the usu­al spikes and dips that occur on dif­fer­ent days of the week and at dif­fer­ent times of the day.

6. Select a testing tool

There are loads of test­ing tools on the mar­ket to choose from, includ­ing Google Opti­mize 360, AB Tasty, VWO, Adobe Tar­get, Opti­mize­ly, and Ora­cle Maxymiser.

7. Launch the advert or email

This is the time when you can sit back or focus on oth­er tasks. Your test will run, mea­sur­ing each inter­ac­tion and col­lect­ing the data.

8. Look at your results

An A/B test­ing soft­ware will present the col­lect­ed data so you can analyse the results of your test. Using a tool will be a big time saver as it auto­mates your test cal­cu­la­tions so you can stick to read­ing the results.

If there are sta­tis­ti­cal­ly sig­nif­i­cant dif­fer­ences between your test­ed vari­ables, these will be inter­est­ing areas to take note of. If your hypoth­e­sis has been con­firmed – con­grat­u­la­tions! You can now move for­ward with confidence.

Next, you can seg­ment your audi­ence for a deep­er look at the data. Seg­ment by traf­fic source, vis­i­tor type, or device type to under­stand how these areas respond­ed to your two variants.

After this is done, you can repeat­ed­ly test dif­fer­ent ele­ments of your mar­ket­ing, sales, and advertising.

Our top tips for conducting A/B testing

1. You don’t have to test everything

Focus on the things that can have the biggest impact on results, such as:

• CTAs
• Head­lines and sub­ject lines
• Sales copy
• Images and graphics
• Audi­ence segments
• Dis­counts and spe­cial offers

2. Divide your segments equally

Divide traf­fic both equal­ly and ran­dom­ly, so there’s no bias. Test both vari­ables at the same time, as the tests might not pro­duce pre­cise results if they’re con­duct­ed at dif­fer­ent times.

3. Don’t test for too long or for not long enough

It’s rec­om­mend­ed that you run tests for about two weeks. A test run­ning for two days won’t pro­duce thor­ough results, as its results won’t allow for the spikes and dips that nat­u­ral­ly occur from Mon­days to Sundays.

Busi­ness­es with small­er mail­ing lists may need to run their tests for longer than two weeks – it all depends on your traf­fic as this will dic­tate whether your results are sta­tis­ti­cal­ly significant.

To fig­ure out if your sam­ple size will be sta­tis­ti­cal­ly sig­nif­i­cant, you can use this free calculator.

Final thoughts

There’s no doubt that A/B test­ing allows busi­ness­es and mar­keters to refine their web­site, advert, and email designs. Although it’s extreme­ly effec­tive at mea­sur­ing tan­gi­ble fac­tors, split test­ing won’t help you under­stand aspects of user expe­ri­ence such as frus­trat­ing ele­ments and ease of use. You’d need to con­sult oth­er chan­nels for this.

If your web­site is func­tion­ing as it should, you have enough fund­ing, and you’d like to revamp your mar­ket­ing meth­ods, split test­ing is def­i­nite­ly a viable route.

Just make sure you’re giv­ing your tests enough time and resources to be reli­able and valu­able. If you do this cor­rect­ly, you’ll have the mag­ic ingre­di­ents to nur­ture leads into return­ing customers.