It is common requirement for marketers to fulfil ‘test and learn’ on campaigns. This could be a simple A/B test of a subject line, or an image test for the front page of a brochure - the purpose of the tests being to understand your recipients’ reactions. For example, if one group is more positive and engaged with you or, even better, fulfils the objective of the campaign - i.e. by making a purchase or leaving a positive review - you might conclude that this is because of the image or subject line received.
To fulfil the test, you need to create a random split of your audience using the %Percentage selection located within the Others section of Audiences in the Library.
- Drag and drop the %Percentage selection into the Content Step to create a Content Field
- Amend the default 50/50 percentage split by clicking on the % icon top left of the selection
If testing more granular splits than 50/50, you can add further percentage splits into the same Content Field by dragging on the Percentage item again. The Other selection appears by default to catch any individuals who do not fall into the preceding segments.
Here 50% of the audience will be randomly allocated to Offer A and the Other selection will cater for the remaining 50% who will receive Offer B.
Automated Alternatives is an extension of A/B testing and is used with subject line or offer tests. It can only be used when you apply an Email channel to the Delivery step and a %Percentage split to the Content step with several alternatives specified. To enable Automated Alternatives:
- Drag and drop a %Percentage split onto your Content step
- Add the required number of splits for the test
In the example below, there are 4 subject lines to test – 3 have been added via the %Percentage selection and the final split is the auto-generated Other selection.
- Select the Automated Alternatives check box in the top right corner of the Content Field
By selecting Automated Alternatives, any defined % splits disappear because – after the initial run - the Automated Alternative process decides the % of the audience receiving each alternative.
The Automated Alternatives checkbox is not visible when you use selections to split the Content. In this circumstance, the selection criteria will constrain the volumes being selected for each subject line and prevent the Automated Alternative algorithm from executing.
Automated Alternative Processing
PeopleStage monitors the performance of each alternative and adjusts the volume of each, using a Champion Challenger algorithm (like a ‘winner stays’ on pool competition).
The initial run of the message is sent to all the alternatives and the best two then become the champion challenger pair in the first test; the other alternatives will take it in turn to be the challenger in subsequent tests.
The champion is determined by analysing the total number of clicks on each version of the email and, depending on the volumes of clicks, the “best” two are selected for the next run of the campaign.
Subsequent tests determine the required volume of communications that need to be sent to make the test statistically significant. Each test sends 80% of communications to the champion and 20% to a single challenger. Once this volume has been sent, possibly over multiple runs of the campaign, the response rates are examined. Only if the challenger has done statistically better will they become the new champion.
Automated Alternatives is effective for regularly run campaigns. In order to determine robust/significant results in response rates, it requires significant volumes to run through the campaign, or that the campaign runs for a long period of time to determine the winning alternative.
To truly determine the success of one alternative over another, the test must remain unchanged, and any additional content/copy components within the email should be static, to mitigate any other factors that might cause a change in a recipient’s decision to click on the email.