What A/B Testing Is
An A/B test (or split test) sends two versions of an email to separate, random groups of subscribers, then compares results. Version A might use one subject line and version B another; everything else stays the same.
Some tools also let you send the winning version to the rest of your list automatically, so most subscribers get the better email.
What to Test First
| Element | Example test | Measure |
|---|---|---|
| Subject line | Question vs statement | Clicks, with opens as a secondary signal |
| Offer | Free shipping vs a percentage discount | Purchases |
| Call to action | "See the Guide" vs "Get the Checklist" | Clicks |
| Email length | Short vs longer with more detail | Clicks |
| Layout | Designed vs simple, near-plain style | Clicks |
| Send time | Morning vs afternoon | Clicks |
| Sender name | Business name vs person at business | Opens and clicks |
Start with subject lines and calls to action. They're easy to change and affect every email.
How to Run a Fair Test
- Write a question, such as "Do questions in subject lines get more clicks than statements?"
- Change one thing only. If you change the subject line and the button, you won't know which made the difference.
- Split randomly. Let your email tool divide subscribers so the groups are comparable.
- Pick your measure before sending, such as click rate.
- Wait long enough. Give the test at least several hours, ideally a day for clicks and longer for purchases.
- Record the result in a simple log so you build knowledge over time.
Sample Size: What Small Lists Can Do
Statistics guides often recommend thousands of subscribers per version for confident results. Many small businesses don't have that, and that's fine.
If your list is small:
- Test big differences, such as two very different offers, not tiny wording tweaks.
- Repeat tests across several emails before drawing conclusions.
- Look for consistent patterns rather than trusting one result.
- Use an online sample size calculator if you want to know how confident you can be.
A small, repeated test that points the same way three times is more useful than a single test you over-interpret.
Reading Results Honestly
Prefer clicks over opens. Privacy features in some email apps can record opens that didn't happen, so opens are a weaker signal than they used to be. See email open rate.
Ignore tiny differences. A gap of a fraction of a percentage point on a small list is usually noise.
Watch the full journey. A subject line that wins on clicks but leads to more unsubscribes isn't a real win.
Apply what you learn. Use winning ideas in future emails, then test the next question.
A Testing Plan for Your First Three Months
Subject lines: questions vs statements, numbers vs no numbers.
Calls to action: wording and button vs text link.
Content: short vs long, designed vs simple.
A/B Testing in AWeber
AWeber's split testing lets you test subject lines, preview text, content, design, calls to action and send times across two groups, or across two groups with a third that receives the winner. You can target a specific segment and view performance for each version.
Split testing isn't included on the free plan; it's available on AWeber Plus. See AWeber Lite vs Plus.
Use AWeber Plus split tests to compare subject lines, content and send times, with a winner sent to the rest of your list.
A/B Testing FAQs
What should I A/B test first in emails?
Subject lines and calls to action. They're easy to change and affect every email.
How many subscribers do I need for an A/B test?
More is better. With a small list, test big differences and repeat tests before drawing conclusions.
How long should an email A/B test run?
At least several hours, and longer if you're measuring purchases.
Does AWeber's free plan include split testing?
No. Split testing is available on AWeber Plus.