How to run an A/B test in PostHog

Write down what changes, who can receive it, which outcome will decide the result, and how long participants have to complete that outcome. Name any harm that would make you stop the test.

Illustrative pricing-page test, not client data:

In our form-quality experiments, more submissions came with reports of poorer qualification and attendance. That is a reason to measure qualified attended calls or purchases per original participant, rather than declaring a winner from form completions alone.

For a completed action, choose a Funnel. Exposure is already the first step: define only the events that follow it. To measure exposure → purchase, the purchase event appears once, with a conversion window. (+)

For an acquisition decision, keep non-buyers in the purchase-rate or revenue-per-participant denominator. Average order value among buyers answers a different question. Use our metric-choice guide to choose between them.

What about bounce rate, time on page and scroll depth?

Use engagement to explain the result unless engagement itself is the business outcome. More time on a pricing page could mean interest or difficulty.

• Bounce rate: the default Web analytics definition is a session with one pageview, no autocaptures and less than ten seconds on the page; the duration threshold is configurable. (+) A user-level purchase funnel is not a bounce-rate calculation.
• Time on page: define the timed interval and capture its numeric duration on a named event for the tested page. Use a mean metric and decide how repeat visits aggregate per participant. Specify whether background-tab time counts; elapsed time alone does not establish attention. (+)
• Scroll depth: inspect $pageleave for $prev_pageview_max_content_percentage before using it. (+) For a clearer completion question, capture a defined section-view event and measure exposure → section viewed as a funnel. Percentage depth can change simply because the treatment made the page longer.

Can you launch without adding metrics?

Yes. Metrics affect analysis, so PostHog lets you launch without them and add, modify or remove them later. Define the primary outcome, denominator, conversion window and MDE before launch anyway; choosing them after looking at results makes it easier to select a favorable answer. (+)

A new event can temporarily prevent saving a metric because PostHog normally checks that each referenced event has arrived at least once. Send it from a permitted QA action first, then add the metric. The API also supports a deliberate allow_unknown_events: true override. (+)

Set the minimum detectable effect around the smallest change worth acting on. If a 2% relative lift would not change a decision, do not build your default around detecting 2%. Agree on the value with the experiment owner before launch.

PostHog’s MDE is relative. With an illustrative 10% baseline conversion rate and a 20% MDE, the target difference is 10% × 20% = 2 percentage points: 12% or 8%, not 30%. The built-in default is 30%. (+)

If you set a lower MDE, you are asking the experiment to detect smaller changes. That usually requires more traffic, more conversions, and a longer runtime. A larger MDE lowers the planned sample; smaller useful effects may remain unresolved.

For a funnel, the calculator’s planning approximation is N ≈ 16 × p × (1 − p) / d² per variant, assuming 80% power and 95% confidence, where d = baseline × relative MDE. In the illustrative example, p = 0.10 and d = 0.02: N ≈ 3,600 per variant, or 7,200 total for two variants. (+) At 360 eligible exposures per day in total, that is about 20 days of enrollment, before allowing the last participants their conversion window.

Plan for the buying cycle as well as the sample. Target reached means the calculator’s sample target has been met; it does not mean the treatment won. Keep the chosen MDE and decision rule fixed while collecting the planned evidence.

Set the default MDE for new experiments

In Experiments → Settings → Statistical analysis, set Minimum detectable effect. This default applies to new experiments in the current environment and can be overridden per experiment. (+) Use the slider or enter a whole percentage from 1 to 100. Changing the default requires project Admin access or higher; ask an admin if you cannot edit it. (+)

Tell the team when the shared default changes and record each test’s MDE when comparing results. Override the default when the business question changes: a high-value checkout may justify detecting a smaller lift than a low-traffic layout test.

Choose how the treatment will run before creating the experiment: application code, PostHog’s no-code toolbar, or our page-experiment runner. Code and runner tests use the product-experiment wizard below. For no-code, create and save the experiment in the toolbar’s Experiment tab, then open it in PostHog to configure metrics and exposure. (+)

For a product experiment, open Experiments → New experiment and work through the three wizard steps:

1. Description: enter the experiment name, hypothesis and feature flag key. Use the naming convention before writing code so the dashboard, implementation and experiment log use the same identifiers.
2. Variant rollout: enter the variant keys and intended split. For a two-arm test, our keys are control and test. Keep control tied to the existing production experience.
3. Analytics: configure inclusion criteria, test-account filtering and the metrics from your plan. Click Save as draft.

A new flag is created when you save; its key is generated from the experiment name and can be customized. To reuse one, choose Select existing flag. It must be multivariate with 2–20 variants and cannot already belong to another active experiment. (+) Results use control as the baseline when present, otherwise the first variant. (+)

Before reusing an enabled flag, stop its public experiment delivery and confirm that disabling it will not break another rollout. Saving a draft does not disable an existing flag.

Check the saved draft’s running time

Once the draft is saved, open the gear beside its running-time estimate. Check the inherited MDE and override it if your plan needs a different value. If it is unexpected, confirm the PostHog environment and that you changed the default on the Experiments settings page, not inside one experiment. Use Manual mode with the baseline metric and expected daily exposures across all variants. After launch, Automatic uses the first primary metric’s available results and observed traffic. Use Manual while a usable automatic estimate is unavailable. Both modes share the same MDE setting. (+ / +)

After saving, open Manage release conditions on the underlying flag. Keep these three controls separate: (+)

For an illustrative 40% rollout with a 50/50 split, about 20% of eligible traffic enters each arm; 60% stays outside. We normally keep the split even and use rollout to limit product risk. Increasing total rollout preserves existing assignments; changing weights or adding variants can move people between arms. (+) Under Inclusion criteria, PostHog excludes participants who saw multiple variants by default; First seen variant is the alternative. Set this before launch and investigate reassignment rather than changing inclusion to improve the result. (+)

For a test on one page, run the implementation only on that route and check that the required elements and page state exist before reading the flag. A site-wide flag read followed by a page check can count people who never reached the change.

Choose the matching identity before audience rules

For anonymous or pre-login traffic, choose Device in the flag’s release conditions. It hashes $device_id, which stays stable through identify() on the same browser. The browser SDK supplies it; a backend evaluating the same device-bucketed flag needs that device ID too. (+)

Use user bucketing when participants are reliably identified before evaluation and the same external ID recognizes them across sessions and devices. A person record or an $identify event alone does not prove that continuity. Test returning visits, login, logout and any client-to-server handoff that matters. Device matching does not join separate devices.

Keep the matching identity unchanged once exposure starts. Changing the bucketing method can reassign participants. (+) Restore production conditions only while the flag is disabled, as in the QA sequence below.

We built PostHog Page Experiments as an independent, MIT-licensed browser package. It waits for the page, checks eligibility before reading the flag, applies the selected handler and falls back to the configured default when assignment is unavailable.

Use the Shopify, WordPress or ClickFunnels guide for insertion points, selectors, consent handling and platform-specific QA. Our public demo shows forced variants. Keep the original experience usable when scripts fail, and test above-the-fold changes for visible flicker.

Exposure should mean the participant had a real opportunity to experience the treatment. A pricing comparison below the fold needs a different exposure boundary from an immediately visible headline. Apply the same entry rule to control and treatment, before any downstream action you want to measure.

A normal browser getFeatureFlag() read can send $feature_flag_called. (+) Check the event named in the experiment’s exposure criteria: it can count $feature_flag_called or, for newer experiments, $experiment_exposure. PostHog can create $experiment_exposure at ingestion from the SDK’s flag call; leave its creation to PostHog. Both carry the experiment’s key in $feature_flag and the assigned variant in $feature_flag_response. (+ / +) Use this flag-call exposure path when the read happens at the tested surface for eligible visitors.

When assignment happens earlier, capture a semantic event at the surface and select Edit exposure criteria → Custom event. Put the resolved variant in $feature/<flag-key> on that event; for example, $feature/010-pricing-page-toggle-design is control or test. Outcome events join through the participant and do not need that property. (+)

Keep debug panels and support tools from creating exposure just to inspect assignment. Use cached flag inspection for those reads. Repeated same-variant exposure events do not multiply unique participants; the more serious denominator error is admitting people who never reached the tested experience. (+)

For a subscription test, define who is eligible for each outcome. Someone who started a trial before seeing onboarding cannot provide evidence that the onboarding increased trial starts. Keep the randomized population for the business decision, and label conditional rates such as trial-starter conversion separately. If native metrics cannot express the intended population, build a reconciled readout with the same assignments, exposure and windows.

First force every variant in a safe preview and check what visitors can see and do. With our runner, a configured URL override bypasses flag evaluation; an unknown value falls back to the configured default. A preview that looks correct proves rendering, not real PostHog assignment. Follow the platform guide for the exact preview parameters.

1. Put the validated implementation on the production route with the flag disabled.
2. While the experiment is still a draft, record production release conditions and their order.
3. Put a QA-only condition first, at 100% rollout with an Optional override for one variant.
4. Set every other public condition’s rollout to zero. Matching any condition set can admit a visitor. (+)
5. Then enable and save the flag to test real assignment.
6. Exercise each variant by changing the QA override, without a runner URL override. (+)
7. Use the detailed assignment check or your platform guide.
8. On runner 0.2.0 sites, add experiment-debug to the query string (?experiment-debug on an otherwise empty URL).
9. Inspect the console summary’s source, flagValue and evaluation reason.
10. Check for source: posthog and a configured flagValue; source: url is a forced preview. (+)

• Visible treatment: the rendered version matches the raw assigned variant. A default control shown after false, a timeout or an unknown value is not an assigned control participant.
• Exposure: the configured event carries the correct experiment and variant at the agreed surface. A below-the-fold treatment must not admit someone only because its DOM exists.
• Outcome: complete a permitted test action and trace its event to the same participant after exposure and within the conversion window. Check server events and cross-domain steps too.
• Failure paths: remove a required target, block the SDK and repeat important navigation or form interactions. The page must remain usable, and ineligible visits must not enter through this implementation.

For the illustrative pricing test, trace one browser assigned test: it reaches the comparison, sends pricing_comparison_viewed with $feature/010-pricing-page-toggle-design set to test, then purchases under the same identity within seven days. Count one exposed participant and one purchaser. A browser assigned test that never reaches the comparison stays outside that exposure denominator.

11. In Activity, find the event named by the exposure criteria ($feature_flag_called, $experiment_exposure or your custom event) and inspect its flag key and variant properties.
12. When every variant passes, disable the flag and save.
13. While it is off, delete the QA condition and overrides.
14. Restore the recorded production conditions in their original order, and save again. Leave it disabled until Launch.
15. Define the project’s internal-user filters and enable test-account filtering in the experiment so staff visits do not become production evidence. (+)

An existing flag may already have its browser exposure cached from before launch. Check returning-user behavior; advanced_feature_flags_dedup_per_session: true enables a fresh flag-call event in a new session. (+) An A/A test and assignment audit are useful when the identity or measurement path is new.

With the linked flag still disabled, reconfirm variant keys, matching identity, release conditions, rollout, split, exposure and metrics, then click Launch. Launching activates the linked flag and starts the experiment. (+) In a new private session without QA or variant overrides, verify a real configured assignment, its exposure and outcome, then reload to check that the variant stays stable. Record the start and deployed implementation version.

Open the experiment’s Exposures panel to see unique participants per variant and the percentage split. Compare those counts with eligible traffic reaching the surface. Investigate a sample-ratio mismatch warning, a missing arm or unexplained missing exposures before interpreting the conversion difference. (+)

We agree a purchase-loss budget before a commerce test starts. During the run, we compare each treatment’s observed purchasers with the purchases expected at the concurrent control rate. That gives an operational signal of purchases gained or foregone while evidence accumulates; it is not financial ROI or a statistical stopping rule. Our A/B test cost guide explains that downside.

A safety stop protects the business; it does not prove the other arm won. If treatments stop receiving traffic at different times, benchmark each against control through its own traffic-stop date and allow eligible late purchases to arrive. Do not keep updating a stopped arm against a later control population.

To stop enrollment while outcomes mature, choose Freeze exposure in the actions menu when available: it preserves enrolled users’ variants and metric collection, whereas Pause disables the flag. It requires a running person-based experiment without holdouts or early-access conditions. A large or empty exposure sample, or more than about 5% anonymous participants without person profiles, can block it. Local-only flag evaluation cannot resolve the snapshot cohort. (+)

Check the experiment’s statistical method before interpreting its labels. Bayesian analysis is the default when no other method is configured; environment and experiment settings can change it. (+)

For a higher-is-better metric, Bayesian chance to win is the model’s probability that a treatment’s effect is positive against baseline. It is not the probability that the implementation is correct, that the effect is commercially large enough, or that this is the best of several treatments. Each treatment is compared with baseline independently, without a multiple-comparison correction. The default confidence threshold is 95%, with 90% and 99% available. (+)

Read the effect size and uncertainty interval alongside that probability, then check the primary outcome, guardrails and elapsed conversion window. A large relative lift on a low baseline can still represent few outcomes. Our two-message headline test illustrates that uncertainty; its changed promise also means the result cannot be attributed to brevity alone.

For a fixed-horizon frequentist test, use the planned sample and readout. Repeatedly stopping at the first favorable p-value inflates false positives. PostHog also offers sequential testing for frequentist experiments, with p-values and confidence sequences designed for repeated checking. (+) If warehouse analysis, sequential testing or metric governance determines your platform choice, compare Confidence and PostHog.

At the planned 7,200 exposures, with every seven-day window complete, the gain is 1 percentage point, or 10% relative. A separate 95% normal approximation for the difference gives roughly −0.4 to +2.4 percentage points; this calculation is illustrative, not PostHog output. It spans harm and the planned worthwhile gain of 2 points. Keep control and record an inconclusive result, rather than shipping on the point estimate or declaring no effect.

Use metric breakdowns when a segment could change the decision, and recordings to investigate a specific behavior. A promising subgroup found after trying several properties is a follow-up hypothesis. In our selector experiment, cumulative variants also required different comparisons: the full package against control could not establish the effect of one added instruction.

Our trial-and-pricing experiment shows why an early signup leader does not settle a mature-revenue decision. Wait for the planned observation window and retain the original assignment when reconciling payments, renewals and refunds.

Record the primary result, uncertainty, guardrails, exclusions, conversion window and reason for the decision. Keep the conclusion separate from whether it has actually been deployed.

If the flag remains responsible for delivery, End experiment lets you serve the chosen variant at 100% within the existing release conditions by default. An all-user rollout is a separate option, so review the target population before confirming. (+)

For a page or theme experiment, we move the approved experience into the source that normally owns it, verify it live, and then remove that experiment’s flag reads, losing elements, preview parameters and temporary styles. On Shopify, prepare that permanent result from the current live theme so later fixes are preserved. Stop flag delivery only after the permanent result works.

When several validated changes have been deployed, use the cumulative-impact guide to assess the programme; adding individual percentage lifts does not give its total effect.