A shorter headline with a more specific promise raised purchases from 0.20% to 0.43% across 7,920 offer-page users over 45 days, with a 96.05% historical win probability. About 25 purchases left a wide relative-lift interval, so use the rewrite as a result for that funnel, not a forecast. An earlier six-message exploration produced a candidate to retest. For your own headline test, decide whether you are comparing complete messages or isolating one wording choice, then measure the business outcome.
Case 1: exploring six headline and subheadline combinations
Visitors registered for an online event, then reached a thank-you page selling a separate low-ticket course. The video explained the offer, but the headline had to give them a reason to watch. We compared control with five variations in benefit breadth, curiosity, speed and supporting copy.
| Arm | Headline approach | Supporting line |
|---|---|---|
| Control | Broad benefit statement with a dated promise | Reassurance about the effort or lifestyle changes required |
| Group 1 | Same headline as control | None |
| Group 2 | Question extending the benefit beyond the original use case | Invitation to watch a short explanation |
| Group 3 | Curiosity-led question about gaining benefits in more situations | First-person invitation to hear the explanation |
| Group 4 | Curiosity statement about one action for faster progress | Invitation to watch a short explanation |
| Group 5 | Speed-led statement with a dated promise | Quantified saving claim |
Purchase-proxy rates and the two reported totals
The report summary records 6,792 participants over one day. The saved PostHog result display contains 6,853 exposed users, with 865 checkout visits and 275 purchase-success events; its results are below. The primary metric was labelled purchase success, with checkout visit as the secondary metric. The implementation used a post-checkout page as the purchase proxy.
| Arm | Exposed users | Checkout visits | Purchase-proxy rate | Relative change | Historical 95% relative-lift interval | Probability of being best |
|---|---|---|---|---|---|---|
| Control: broad benefit | 1,047 | 147 | 4.01% | Baseline | Baseline | 5.79% |
| Group 1: no supporting line | 1,190 | 141 | 3.45% | −14.11% | −36.35% to +15.70% | 0.64% |
| Group 2: broader-benefit question | 1,146 | 161 | 5.24% | +30.52% | +2.01% to +66.58% | 72.52% |
| Group 3: curiosity question | 1,151 | 139 | 4.52% | +12.62% | −13.62% to +46.52% | 18.59% |
| Group 4: one-action statement | 1,162 | 131 | 3.79% | −5.61% | −29.30% to +25.79% | 2.33% |
| Group 5: speed-led statement | 1,157 | 146 | 3.11% | −22.43% | −43.67% to +6.65% | 0.13% |
The saved display labels Group 2’s 72.52% the “probability of being best” and shows a significance banner. That banner does not validate a win. PostHog’s current Bayesian “chance to win” compares a treatment with control; it is a different quantity from the historical best-arm label. (+ / +)
The report and display have different totals; their observation windows and statistical settings have not been reconciled.
Retest Group 2’s broader-benefit question against control with a verified purchase event before extending it to other pages.
Case 2: a shorter headline with a different promise
Later, on an offer page in the same offer family, we replaced a long question with a shorter directive focused on one outcome and added a dated promise. We changed only the headline on the page, splitting traffic 50/50.
| Dimension | Control | Treatment |
|---|---|---|
| Structure | Question | Directive |
| Length | Longer | Shorter |
| Benefit | Several possible rewards | One specific long-term outcome |
| Time framing | None | Dated promise |
This was one page element, but several copy decisions: question versus directive, broad versus specific benefit, length and a dated promise. The result belongs to that complete message.
Can you reuse a dated headline?
Before reusing a dated promise, check that the offer can support the new date and the outcome being promised.
The 45-day purchase result
Across 7,920 users over 45 days, the displayed rates differ by 0.23 percentage points. The historical relative-lift estimate is +110.05%, with an approximately +31.88% to +235.79% displayed relative-lift interval. Only about 25 purchases occurred, so the direction is clearer than the size of the gain.
| Arm | Purchase rate | Historical relative lift | Win probability |
|---|---|---|---|
| Control | 0.20% | Baseline | 3.95% |
| Treatment | 0.43% | +110.05% | 96.05% |
The historical 96.05% win probability supports the rewrite in this funnel, rather than predicting the same lift on another page.
How to plan your headline test
What should you change in a headline test?
- To explore message angles, compare complete packages. The six-way case varied benefit breadth, curiosity and supporting copy.
- To isolate one choice, keep the rest of the message fixed. A test of headline length should preserve the benefit; the two-arm case changed both and cannot tell us that shorter wording alone helped.
Which metric should decide a headline test?
Choose purchases or qualified leads when that is the page’s business goal. Use video starts and checkout visits to explain the journey, rather than replacing a weaker purchase result. In the six-way case, the purchase metric was a page-view proxy; a retest should verify payment completion.
How many headlines should you test, and for how long?
Use control plus one focused rewrite when you have a clear hypothesis. Add more message packages when the question is exploratory and traffic can support them. One day produced a six-way candidate; 45 days still left the two-arm result with only about 25 purchases. Plan from your baseline, the smallest lift worth detecting and expected daily exposures across all variants. PostHog’s manual calculator uses the metric type and those inputs to estimate sample size and runtime. (+)
Set the duration, sample target, conversion window and decision rule before launch. Reaching a sample estimate and having enough evidence to choose a winner are separate checks; do not stop a multi-way exploration simply because one message leads on the first day.
Our relatability test examines copy built from applicants’ language; the event start-time test isolates one scheduling detail in a headline.
How do you run the comparison in PostHog?
Use our PostHog experiment guide for setup and the metrics guide to connect a headline’s early clicks with the business outcome.