How to Test Push Notifications for UX and Engagement
Sending push notifications is not an exact science. Even though your push notification campaign should be driven by cutting-edge mobile engagement and data analysis tools, there are so many variables that it may seem like more of an art – or even magic! However, the savvy mobile marketer knows that a successful push campaign is the result of extensive research, segmentation of users, and A/B testing. This can seem overwhelming at first glance, so let’s break down how to test push notifications for UX and engagement.
Download our FREE Mobile Marketing Playbook to perfect your user engagement game!
What Is A/B Testing for Push Notifications?
A/B testing is basically when you take the scientific method to different variations of a push notification. For example, say you have a cohort of 10,000 users you want to send a notification to. You would randomly section off 1,500 of those users to act as guinea pigs for 3 different versions of the notification you want to test. Now, let’s say you want to inform these 10,000 users of a sale in flights to New York City using your travel app. Your 3 notification templates might go something like:
Hi [user name]! Are you ready to see the city lights? Book a roundtrip flight to NYC for only $180, this weekend only! ✈️
2020 has been a tough year, [user name]. Check out our flash sale this weekend and reward yourself with a roundtrip flight to NYC for $180!
[user name], we want to see you living it up in the Big Apple! 🍎 Book your roundtrip flight to NYC for $180!
You would then send each of these notifications to a test group of 1,500 users. After waiting a certain amount of time (also variable… you can wait anywhere from 1 to 24 hours for your final results), you would analyze which version got the best rate of app opens. That is the notification you’d then send to the remaining 5,500 users.
Of course, the wording of the notification is only one aspect you can A/B test for. You may also wish to test according to what time of day receives greatest click-through rate (CTR). Or you may wish to test your push notifications by frequency. Or according to any of the 40+ data signals that OpenBack can use to personalize your notifications.
The best thing about A/B testing is that the results can be aggregated over the course of your app’s lifespan. The more testing you do, the better your push campaigns will get overall.
What Types of Variables Can You Test For?
The variables for push notifications are practically endless. Above we looked at different variations on content.
In a similar vein, you may wish to test for different types of tone with which you speak to users. This can include whether or not to use humor – and what type of humor? Puns, yes. Sarcasm? Maybe not so much. It all really depends on what the context of your app is, as well as user demographics. For example, take this push notification stunt by the gaming app “Simon Circles.” It was so on the money that mobile marketers are still talking about it 6 years later:
In fact, studies have shown that humor is a good way of getting your brand to resonate with Millennial users. And this opens up other forms of multimedia that Millennials and Gen Z users are fluent in, such as GIFs, memes, and emojis. However, older generations or users of an app with less scope for humor – say, a financial planning app – may not appreciate your witty Grumpy Cat quips.
Timing is also incredibly important to test for. “Adaptive scheduling,” or the concept of sending notifications at the moment that’s most convenient for users to read them, has been shown to boost CTR by an average of 23.3%. And users’ devices provide a wealth of data to inform your push campaigns about the best times to send notifications. By leveraging the information in users calendars, work schedules, and even just their habitual daily device usage, you can block out windows of time that are most likely to achieve results.
For example, for many people the first thing they do when they wake up is scroll through their phone. So if a user consistently has a block of phone usage from 7 to 7:30 am, followed by patchy use until 1 pm, when they have another half-hour block of scrolling, it’s likely that person works a standard 9-to-5 workdays. Keep that in mind when sending notifications – it’s more respectful of users’ time, and will likely get more engagement as a result.
However, it’s imperative to test timing of different push notifications to make sure you get it exactly right for example, you might segment your users based on the following scheduling demographics:
- 9-to-5 work
- Night shift work
- Freelancer or college student
For each of the segments, you can run A/B testing based on different times of the day. Once you have the wording of your notification just right, send it to 3 different test groups: try 11 am, 1:30 pm, and 7:30 pm. For a user who works 9 to 5, each of these are presumably times they might have the leisure to look at their phones. OpenBack’s machine learning features But testing with give you the stats to show you which time gets the highest CTR for sure.
Other Data Signals
OpenBack is able to leverage hundreds of different data points to optimize your push campaign. They have all the standard signals such as geolocation, past purchases, and other digital behaviors. OpenBack can also tell you:
- how much battery power users have
- whether their phone is in use
- if the lockscreen is engaged
- whether the headphone jack is in use
- and much more
Whether to use some of them is common sense – don’t waste time sending a push notification with a video clip to a user with only 2% battery life. Even if they do start viewing the video, their phone might die in the middle of it, and they likely won’t seek it out once they start their phone up again. But other data signals may benefit from A/B testing to tease out the subtle benefits of sending push notifications informed by different combinations of signals.
With access to these signals, OpenBack is able to provide a truly personalized user experience in a way that other push notification SDKs fall short of.
For more information on what makes OpenBack unique, read our blog post: 7 Features Distinguishing OpenBack from Other Push Notification Platforms
Test Push Notifications With OpenBack’s Other Metrics
In addition to offering A/B testing, OpenBack provides multiple metrics to analyze your results by, beyond simply whether your user clicked on the app or not. Any action – or lack of action, for that matter – a user takes following delivery of your push notification can serve as valuable market research on how to craft the perfect push campaign.
The OpenBack dashboard offers full push notification metrics and analytics, with complete message campaign data that include:
- confirmation that the notification has delivered
- whether the user has interacted with notification
- whether they dismissed or clicked on the notification
- if they went into the app
- whether the notification resulted in a goal completed
- whether the notification resulted in an app uninstall
OpenBack also offers event tracking, cohorts, time spent in the app, and churn prediction for various segments of users. OpenBack lets you track different goals and whether a certain push notification directly resulted in conversions. On the other hand, it also shows whether a certain notification resulted in the user opting out of notifications. Or even uninstalling the app.
Benefits of Machine Learning
Another unique feature to OpenBack is its machine-learning driven audience estimation and prediction capability. Our platform can predict the correct target user for a certain notification campaign. It can also predict the number of users likely to engage with that notification based on data signals at the moment of delivery.
With the most comprehensive capacity for data leverage of any push notification platform, OpenBack offers a full perspective of users’ habits and their role in your app ecosystem. Having these push notification testing tools at your disposal, and employing diligent A/B testing for your push campaigns, you are sure to maximize click-through rate and boost revenues.
Calculate how much your revenue would increase per month using OpenBack: