Push Notification A/B Testing 101

Push notification A/B testing is a secret weapon for your push campaigns. 

We absolutely love push notifications around here, and so do our users, who send hundreds of thousands of push notifications every month through their MobiLoud apps to:

• Drive engagement
• Recover abandoned carts
• Promote hot offers

And much more!

But push notifications must be used tactically. 

You can just fire them off into the void and hope for the best, that will probably work somewhat.

But, you’ll get much better results by constantly measuring, testing and optimizing your push campaigns. 

This is much easier than it sounds. A/B testing push notifications doesn’t need any kind of advanced tools, technical set-up, or arcane statistical knowledge.

We’re going to teach you the 80/20 of push notification A/B testing in this article. We’ll show you why you need to test, what to test, and how to get the whole thing set up and running on autopilot. 

After you’ve implemented this and acted on the data you gather, your push notification campaigns will skyrocket in performance in 2024 and beyond. 

Let’s start by briefly covering the big picture. 

What is A/B Testing?  

A/B testing is a crucial part of the conversion rate optimization toolbox, and is a scientific, experimental process.

Traditionally it means testing two versions of a web page, with one element different in each. The original page is the control, while the altered page is the variation. 

Optimizers will create hypotheses predicting that certain changes to the page will affect metrics like dwell time, CTR, or conversions. 

By randomly sending traffic to both versions and measuring how users interact with them, the hypothesis can be tested, and the results used in future, higher performing designs. 

The typical workflow for A/B testing is:

1. Collecting performance data to find optimization opportunities
2. Building a hypothesis to propose changes to elements
3. Creating variations to test against a control 
4. Using A/B testing tools to run the experiment & analyze results
5. Collecting further data during to design the next tests & iterations

Basically, A/B testing allows you to make data-driven decisions. It's a proven method that produces great results. 

Now let’s move on to push notification A/B testing specifically. 

A/B Testing Push Notifications 

So we know what A/B testing is in general, so what about push notifications? 

The overall concept is the same, we form hypotheses about what elements of a notification will improve performance, then we create variations to test whether our assumptions are correct. 

We use tools, typically provided by our push notification platform, to run the experiments and measure the results. Then we implement what we learn into future campaigns and come up with new hypotheses to test.  

The main difference is that there are simply fewer elements we can change for push notifications compared to emails, landing pages, and ads. 

This ultimately makes testing and optimizing easier, and we’ll cover an exhaustive list of what to test shortly. 

But just because push notification A/B testing is simpler, doesn’t mean it isn’t super powerful. 

What Tools do you need to A/B Test Push Notifications? 

Generally, solid push notification service providers like:

• OneSignal
• Airship
• PushEngage

Give you all kinds of tools for managing push notifications - including A/B testing features. 

So there should be no need for you to use separate tools just for testing push, and almost definitely no need for any kind of custom implementation. 

OneSignal, for example, is the platform we integrate with - allowing you to send unlimited push notifications to your users from your MobiLoud iOS and Android apps. 

Whatever features OneSignal supports, you can set up for your MobiLoud apps. 

In the case of A/B testing, OneSignal has you covered completely. 

OneSignal allows you to A/B test up to 10 message variants on the Pro plan and even gives you A/B testing features on their Free plan! You can track everything simply through their native dashboards.

If you use another push notification provider, check out their docs. All the market leaders will have similar A/B testing features with minor variations in functionality and pricing. 

So there’s no need to worry about anything too “technical”, the platform you use already built the necessary infrastructure. It’s more important for you to focus on designing intelligent tests. Let’s look at how. 

Push Notification A/B Testing Process

On a high level, here are the important steps for setting up push notification A/B tests. 

1. Goal Setting 

What does success look like? What metrics are you trying to improve or optimize for?

Typically for push, we’re looking for higher CTR or conversions! 

2. Hypothesis formation

Use your marketing intuition, industry best practices, or previous data to come up with a testable guess about the impact of some change. 

It should be simple, and easy to state in a sentence. Like this:

• Notifications with a clear call-to-action, like 'Shop Now', will have a significantly higher click-through rate than notifications with a generic CTA like 'Learn More'
• Incorporating emojis in push notifications will lead to a 20% increase in open rates compared to notifications without emojis.
• "Push notifications containing discount offers will see a 40% higher conversion rate compared to notifications with new product announcements."
• "Reducing the frequency of push notifications to once a week will decrease the unsubscription rate by 15% compared to daily notifications."

The basic format: “If we change x (element) then y (result) will happen”. 

3. Segmentation 

Using the push platform’s tools, you’ll divide users into two or more random, yet statistically similar groups to ensure a fair and unbiased test. 

This is fundamental to ensure that the outcomes of the test are relevant to your hypothesis - and  not due to underlying differences in the audience groups.

4. Variant Design 

Here you’ll actually develop the two (or more) variants with only one key difference - the variable that your hypothesis rests on. 

Examples are the message content, the call to action, or the timing of the notification.

This sole focus on the single variable is crucial for drawing clear conclusions from the test. 

5. Test Duration / Timing 

Finally, decide how long the test will run and when. 

The test must run long enough to gather a meaningful amount of data but not so long that random events and factors can skew the results. 

So that is the process of how to set up your push notification A/B tests. Now let’s take a look at some good candidates for testing. 

What should you test? 

Push notifications are their own channel, with their own particular quirks and differences. Your testing focus should reflect this. 

The most important factor: push notifications are immediate. 

They’re designed for instant interaction, and so live and die by how effective they are at enticing the user to click as soon as they see them pop up on their lock-screen. 

That’s why open rates and click through rates are the most common metrics to optimize for. 

OS Specificity is also important for push notification A/B testing. An email inbox is an email inbox, a browser is (sort of) a browser - but notification UX is very different on iOS and Android. Make sure your tests take that into account. 

Push notifications are also concise. There’s limited room for text - only a few lines and <200 characters - so testing punchy copy variations for headlines and CTAs can be instructive. 

7 Variables To Test 

Here are a few suggestions for variables to build hypotheses around. 

• Frequency: push notifications are known for “fatiguing” users if they’re bombarded with them too much. Testing to explore the optimal notification frequency to engage users without annoying them is a good idea!

• CTA: Experiment with different CTAs to see which leads to higher click-through rates. Test the wording of the CTA, its positioning, its style, and anything else that could make an impact.

• Copy Style: test different versions of the notification text to see which hits hardest. Try varying the tone (formal vs. casual), use of emotive language, clarity and conciseness, and use of psychological triggers like scarcity and urgency.

• Media: the classic push notifications are text only but you can also test images, video, and audio to see how they perform versus text only.

• Timing: since push notifications are immediate, the time of day or day of the week that your users receive them can have a huge impact on their engagement, test it out and see how they react!

• Personalization: test the impact of personalized messages by using their name or referencing past behavior versus generic messages to see how the personal touch impacts target metrics.

• Length: Test short versus long notifications to determine the optimal length.

These seven potential variables should keep you occupied for quite some time, and optimizing all of them will dramatically improve the results of your push campaigns. 

Push Notification A/B Test Tips 

Before we wrap up, a summary of the most important points and our top tips. 

Important: focus on testing one thing at a time. 

You should focus on single variables per test to make sure your results are valid. Once you’ve proven (or disproven) your hypothesis, then you can move onto the next one. 

This makes it simple and manageable for small businesses without large teams of optimizers. 

It’s also crucial to get a large enough sample. 

The groups you test on need to be large enough for statistical significance, so that you can be confident that the results are not from mere random chance. 

You can read more about sample size and statistical significance in this primer. On a practical level though it means don’t worry so much about testing until you have a significant (thousands) number of users receiving your notifications. 

Until then focus on best practices and acquiring more users!

A/B Testing your Push Notifications with MobiLoud

So we’ve seen that push notification A/B testing is generally as simple as: 

• Reading the docs of your push notification provider
• Making hypotheses to test 
• Running experiments 
• Implementing results and making new hypotheses in an iterative loop 

MobiLoud can make this easier for you in several ways. 

Firstly, MobiLoud converts your website into native apps for iOS and Android. 

We build you high-end apps, handling the entire development process, and we integrate everything from your existing site. 

We've built thousands of apps for countless ecommerce brands - from small startups to multibillion dollar multinationals.

Your MobiLoud apps come pre-integrated with OneSignal, the best push provider on the market. 

You can send unlimited push notifications to your customers through OneSignal, and also make use of OneSignal’s push notification A/B testing features. 

We can help you get set up with this as part of our full service where we handle all aspects of the app build for you, including ongoing updates and maintenance. 

We also have our own system for abandoned cart notifications. 

These are crucial for eCommerce, and we’ve created a system where an abandoned cart notification sequence is automatically triggered when we detect items in a users’ cart (when the app is closed). 

We use best practices from CRO and powerful copywriting for the notification sequence, and test the results to optimize continually. 

So if you’re an eCommerce brand who wants to build mobile apps, abandoned cart notifications are all automated and handled by our team for you, so you can focus on testing and optimizing other kinds of notifications like promotions and welcome sequences. 

MobiLoud also integrates everything from your site onto the apps. So whatever web tools you use for advanced analytics or landing page A/B testing will work in the apps too. 

You can literally transfer everything great about your web store into the apps!

Get Started with Push Notification A/B Testing

You now know enough to get started with push notification A/B testing. 

You can get started with the web immediately, and if you’re ready to start building eCommerce apps for iOS and Android which will skyrocket your businesses core metrics, we’re ready to explain how MobiLoud will get you there. 

It starts with a quick chat with one of our app experts. 

Book a demo call today!