How to Set Up PostHog A/B Testing on Shopify

If PostHog is not installed on your storefront yet, follow the Shopify installation and event-tracking guide first. If you are still choosing an installation method, compare the Shopify PostHog integration methods.

• A working PostHog project and its browser client available as window.posthog on the tested pages.
• Shopify access to duplicate and edit themes, with someone responsible for reviewing and publishing the finished test.
• A page or template to test and a downstream business event already arriving in PostHog.

For a purchase metric, checkout events must belong to the same PostHog visitor as the storefront exposure. (+) A custom pixel sending purchases under an unlinked identity can leave the experiment with zero purchases even while those events arrive. Verify the identity bridge between storefront and checkout from our install guide before building the test.

Write the test as a simple hypothesis:

Changing [element] from [control] to [variant] will improve [metric] because [reason].

For example: changing a long product description into key points will improve purchases because shoppers can find the information they need more easily.

Change what the hypothesis calls for. One change makes it easier to explain the result. You can test several related changes as a package, but then the result applies to the package; it does not tell you which part caused it.

Agree on the feature flag key before coding, then use that exact key as the experiment slug in Shopify. For example:

Experiment: 026 - Product Page - Description Key Points
Feature flag: 026-product-page-description-key-points
Variations: control, test

For three or more variations, use stable machine keys such as control, test_group_1, and test_group_2. Descriptive labels belong in PostHog; the machine keys should remain short and unchanged after exposure begins.

Never implement experiment code directly on the live theme. The duplicate gives you a safe place to build and preview the test without changing what shoppers see.

In Shopify admin, open Online Store → Themes.

1. Identify the current live theme.
2. Open its actions menu (the three dots) and select Duplicate.
3. Give the duplicate a clear experiment name and record its theme ID.
4. Open Edit theme on the duplicate and confirm that it is unpublished.

You can reuse an existing experiment theme only after confirming that it is unpublished and still contains the intended baseline. Before changing a theme file or Custom HTML block, read its current contents and preserve unrelated code. Keep experimental copy in the theme: a duplicate theme does not create separate product records, so editing a product description in admin changes it for every visitor. (+)

Put the code where it will run on the pages you want to test:

• Product-page experiment: reuse the product template’s existing Custom HTML or Custom Liquid experiment container when it has one.
• Other template-specific experiment: use the corresponding container on that template.
• Site-wide experiment: use the header or footer according to where the change belongs. Use theme.liquid only when the code must run globally or before those sections render.

If there is no container, open the duplicate in Edit theme, choose the tested template from the Home page drop-down, then select Add section. (+) Choose Custom Liquid, or your theme’s Custom HTML equivalent, paste the script from Step 5 into its code field, and save. Section choices depend on the theme; ask a developer to add a suitable section if neither is available.

Keep the script-only section enabled so its code runs, but hide the empty space it adds. Inspect its outer shopify-section-… wrapper in the storefront, then use that exact ID with display: none; in the theme stylesheet or Theme settings → Custom CSS. Section-level Custom CSS cannot target the outer Shopify section wrapper. (+) Check that the script still loads and the section leaves no gap.

For small text, CSS or HTML edits on a server-rendered page, you can use PostHog’s no-code web experiments: add disable_web_experiments: false to the snippet and build the change in the toolbar. Use the runner below for changes to the page structure, checks on who can receive a variant, or repeatable URL previews. Either approach needs extra code if the theme later replaces the edited elements.

Do not assume that one product template, section ID, or selector represents the entire storefront.

1. List every Shopify template and template suffix where the experiment should run.
2. Find the relevant section, block, code setting, or target element in each template.
3. Open a representative storefront URL for every affected template.
4. Compare the IDs in the theme source with the IDs rendered in the browser DOM.
5. Confirm that every selector matches the intended element exactly once.
6. Record template-specific selector or mount differences instead of forcing one selector everywhere.

If the elements have no stable selectors, add data-experiment-control to the existing control container and data-experiment-test to the treatment container in the Liquid section, block or snippet that renders them, or ask a developer to do it. Keep the treatment markup beside the control on the tested template; the script-only container from Step 3 is a separate section.

Watch for Shopify’s dynamic ID suffixes

Shopify assigns dynamic IDs to sections in JSON templates. (+) Compare the rendered ID with the source rather than guessing its format. If your target changes from Block-description to Block-description__abc123, the selectors differ as follows:

/* The rendered ID stays unchanged */
#Block-description

/* The rendered ID becomes Block-description__<dynamic-suffix> */
[id^="Block-description__"]

Use the starts-with selector only after confirming that the theme actually generates the suffix. When more than one element could match, scope the selector to the nearest stable container.

Include markets in the template list. Market-specific sections and blocks can change the layout; different translations or currencies may therefore need separate selector mappings or separate experiments when the intended experience differs. (+) Check each supported locale and currency. For pages built with a page builder, inspect the generated DOM and verify selectors again after a builder edit.

Swap tests and existence tests need different control logic

• Swap test: the existing control element and treatment element both have selectors.
• Existence test: control preserves the existing storefront and may not need its own selector. The treatment introduces or reveals the new element.
• DOM-mutation test: control leaves the current element unchanged while the treatment changes its copy, attributes, order, or behavior.

If control leaves the page as it is, you do not need to create an extra element just for the script.

The example below swaps a product-description block for key points. It assumes both containers are present when the page becomes ready and that this is the only experiment using these example attributes on the template.

First, hide the treatment by default so visitors cannot see both versions before the flag loads. Add this CSS to the template’s stylesheet, or inside a <style> tag in Custom Liquid, before the treatment can paint:

[data-experiment-test] {
  display: none !important;
}

Leave control visible. A swap after the flag loads can still move the layout, so reserve the required space or test the layout shift. In the script, change display: 'block' to flex or grid if that is the treatment container’s intended display type.

Load the runner pinned to version 0.2.0 once, then add the call below to the experiment container. It waits for DOM readiness, checks eligibility, reads the URL override or PostHog flag, and applies the selected variant.

<script src="https://cdn.jsdelivr.net/npm/@99ways/[email protected]/dist/posthog-page-experiments.min.js"></script>
<script>
window.runExperiment('026-product-page-description-key-points', {
  isEligible: async () => {
    // A URL preview of a variant reads no flag; every other visit waits for consent.
    if (!new URLSearchParams(location.search).get('026-product-page-description-key-points')) {
      await window.analyticsConsent
      if (window.posthog?.has_opted_out_capturing?.()) return false
    }
    return ["[data-experiment-control]", "[data-experiment-test]"].every(selector =>
      document.querySelectorAll(selector).length === 1
    )
  },

  variants: {
    control: [],
    test: [
      {
        selector: '[data-experiment-control]',
        updates: { style: { display: 'none' } },
      },
      {
        selector: '[data-experiment-test]',
        updates: { style: { display: 'block' } },
      },
    ],
  },
}).catch(error => console.error('Experiment error:', error))
</script>

Replace the example key, in both places, and the selectors with your verified values. isEligible checks that both targets exist exactly once before any URL override or flag read. Without it, the runner accepts visitors by default and skips selectors that match nothing, which can record exposure without applying the treatment. A false eligibility result skips assignment and returns null. The runner applies configured styles with !important.

isEligible first waits for window.analyticsConsent, which the theme code in our install guide resolves once Shopify allows analytics; URL previews skip the wait because they read no flag. A visitor who accepts after the page has loaded joins on that page, instead of seeing control after the runner’s four-second timeout. The test therefore measures visitors who allow analytics, from the moment they do: visitors who never accept see control and stay outside the analysis, and a late acceptor may already have read the original description. If your store loads PostHog opted out instead, declare the promise as our theme code does, before the experiment runs, and call window.allowAnalytics() right after posthog.opt_in_capturing(): a flag read before acceptance drops its exposure, opting in does not resend it, and later reads of the same value can stay suppressed. (+) Without consent gating the value is undefined, and the check continues at once.

The runner applies the variant once. Calling runExperiment() on page load does not make it watch for later changes. If a cart drawer, variant picker or Section Rendering API update replaces those elements, the treatment can disappear. If targets load late, wait for them before assignment, with a time limit. If they are replaced, keep the assigned variant and reapply its changes after the theme finishes updating. Repeating those changes must not duplicate content or apply the same edit twice.

Use the feature flag key as the query-parameter name and a configured variant key as its value:

?026-product-page-description-key-points=test

For an unpublished theme, start with Shopify’s Preview link and preserve its existing parameters. Combine the variant parameter with preview_theme_id when that is present. This illustrative URL uses a placeholder theme ID:

https://example.com/products/example-product?preview_theme_id=123456789&026-product-page-description-key-points=test

Change test to control to check the original experience. Eligibility runs first; a forced value then bypasses the PostHog flag read. An unknown, nonempty value also bypasses PostHog and shows the configured default, so use only your exact variant keys.

Create a complete preview URL for control and every treatment on every affected template. For reviewers, use Shopify’s share-preview control and append the variant parameter to the generated link. Merchant previews require admin authentication and support checkout access; use one for checkout QA. A forced URL checks rendering, not real flag assignment.

Open the browser console. Version 0.2.0 logs an assignment summary containing the selected variant, source, raw flag value and, when available, PostHog’s evaluation reason and IDs. A forced preview shows source: url. Add &experiment-debug to that URL for step-by-step logs; use ?experiment-debug on a URL without other parameters. ?experiment-debug=flag-a,flag-b limits debug output to those flags. After QA, drop experiment-debug from your URLs and set log: 'silent' in the runner options for no console output; the URL switch overrides the silent setting.

• Force each variant and confirm that only one version appears on mobile and desktop, with no broken layout.
• Exercise product forms, variant selection, add to cart, cart drawers and checkout entry; check that the treatment survives any section updates.
• Repeat on every affected template, market, locale and currency. For a complex visual change, temporary “Control” and “Test” labels can make the preview unambiguous; remove them before publishing.
• Remove a required treatment target in a safe preview: the example should leave control visible and make no flag call. Block the PostHog load and confirm the storefront remains usable.
• Visit without a forced parameter. Keep the flag disabled during this review and confirm control appears. This tests fallback, not normal assignment. Exclude staff and preview traffic from production analysis; other tracking can still record events on a forced visit.

Once the test works on the unpublished theme, open Experiments → New experiment in PostHog. In Description, enter the name, hypothesis and exact feature flag key. In Variant rollout, use the same variation keys as your theme code.

In the creation wizard’s Analytics step, configure inclusion criteria and add a funnel metric under Primary metrics. For a purchase-conversion test, choose the actual purchase event arriving from your pixel and set a conversion window that covers the buying journey. PostHog automatically adds exposure as the first funnel step; specify only the subsequent events. (+) Click Save as draft after configuration.

Choose Match by Device for normal Shopify storefront traffic

For anonymous or pre-login storefront traffic, use Device matching. After saving the draft, open Manage release conditions and select Device under the flag’s Release conditions. (+) PostHog hashes $device_id, which stays stable across identify() on the same browser or device.

Use User only when the store has a verified persistent account identity, participants are consistently identified before the flag is evaluated, and the same external ID returns them to their original PostHog profile across sessions and devices.

A PostHog person record, distinct_id or $identify event alone does not show that User matching will work. Check how you recognize returning users before choosing it. Device matching keeps assignment stable on one device; it does not connect different devices to the same participant.

Complete the experiment configuration

$feature_flag_called tells you the flag was read, but the changed block may still be below the fold. If the analysis should include only people who saw that area, send a custom event when it becomes visible in either variant. Include $feature/026-product-page-description-key-points with the assigned variant, then select it in Edit exposure criteria → Custom event. (+) Send it only after a real control or test flag response and successful rendering. Apply the same visibility rule to both variants so the comparison counts people at the same point in the page.

For a revenue experiment, we would usually measure purchases. Use the A/B testing metrics guide if your hypothesis calls for another outcome. When purchases are sparse, add-to-cart and checkout-started events can help explain the journey, but an increase in either does not prove a purchase lift.

Keep the variation split stable, usually 50/50 for an A/B test. Use rollout percentage to control release risk instead of changing variation weights. (+) Changing the split can reassign visitors; increasing total rollout preserves the variants of those already assigned.

Before launch, set the minimum detectable effect: the smallest relative change worth detecting. Use PostHog’s running-time calculator in manual mode for drafts, with your baseline metric and eligible daily traffic to plan sample size and duration. Allow at least one full buying cycle as well as the data required for that effect; a fixed number of weeks alone is not a stopping rule.

Share the unpublished theme URLs for review, together with the affected templates, selector mappings and PostHog draft. Recheck any theme changes made since the review before publishing.

1. Compare the duplicate against the current live theme and carry over any later fixes that must ship with it.
2. In Shopify, go to Online Store → Themes, find the tested duplicate under Draft themes and click Publish.
3. Check the public page without a forced variant: product actions, checkout entry and event capture must still work.
4. In PostHog, confirm the flag key, variation keys, split, release conditions, exposure criteria and primary metric, then test one real assignment as below before you launch.

The previous theme remains in Draft themes; publish it again if you need to roll back. (+) Each saved theme keeps its own edits. Publishing the experiment theme will omit any later fixes made only to the old live theme, and rolling back will omit changes made only to the experiment theme. Copy over the changes you need before publishing either one.

If you roll back because the experiment is broken, pause the PostHog experiment as well, so it stops collecting new exposures while you fix the implementation.

Test a Real Assignment Before Launch

Runner URL overrides bypass normal flag evaluation, so before launch, test a real PostHog assignment with restricted release conditions. (+) We run it on the published theme, where the real checkout is available. Record the flag’s production release conditions first. Then open the experiment’s feature flag and add a condition set at the top of the list where $initial_utm_source (Initial UTM source) equals posthog-qa, with its rollout at 100% and its Optional override set to test. Set every other condition’s rollout to 0%: a visitor who matches any condition set receives a variant. (+) Check Enable feature flag and save; the experiment stays a draft. The browser SDK sends the first page’s UTM source with flag requests, so the QA condition matches an anonymous visitor. (+)

Close all private windows, start a new private session and open a tested product page with ?utm_source=posthog-qa&experiment-debug as its first page. The runner’s summary should show source: posthog with the flag value test, and the treatment should be visible. In Activity, find the exposure event the experiment uses, $feature_flag_called, $experiment_exposure or your custom exposure, with this flag’s key and test. (+) Complete a permitted test purchase and trace the purchase event to the same participant, after that exposure and within the conversion window. This purchase stays out of the results, which start at launch. Repeat with the override set to control in a new private session. Finally, in another new private session without the QA parameter, check that the page shows control.

When both arms pass, disable the flag and save. Then delete the QA condition set, restore the recorded production conditions and save again. Click Launch in the experiment header: launching enables the flag and starts the experiment.

If PostHog waits for consent, accept the banner on the product page during this test. Control should show until you accept; then the treatment and its exposure should appear. On a clean, unforced visit, a treatment that appears before acceptance points to an early flag read.

Watch the first visitors after launch

Close all private windows, start a new private session without the forced-variant parameter, add ?experiment-debug and check all three levels:

• Visual: confirm the displayed variant matches the assignment summary. For this multivariate flag, require source: posthog and a flagValue of control or test. A displayed control with flagValue: false is not an assigned control participant. Reload and verify the same device keeps its variant.
• Event: in PostHog Activity, find the configured exposure event for this flag: $feature_flag_called or $experiment_exposure. Check $feature_flag and $feature_flag_response, or check the chosen custom exposure and its variant property.
• Experiment: confirm exposures reach the intended variants and that valid purchases enter the metric. Investigate a sample-ratio mismatch or missing data before interpreting conversion differences.

Before concluding that the treatment performed worse, check the selector, script, metric definition and Shopify event tracking. Keep the metric, content and split stable while collecting the planned data. An early difference based on a few purchases is not enough to choose a winner.

• Control always appears: read the raw flagValue and evaluation reason. An unavailable SDK, a disabled flag and an out-of-rollout visitor can all show control for different reasons.
• Only some products show treatment: compare their template suffixes, target counts and section-update behavior against the URLs that passed QA.
• Exposures arrive but purchases do not: check the event name, identity link, ordering and conversion window before changing the variant.
• Returning visitors are missing from exposures: the web SDK can cache flag-call events across sessions. If the flag was evaluated before launch, consider advanced_feature_flags_dedup_per_session: true in the PostHog initialization so the event can be emitted in a new session. (+)

After checking the measurement, use metric breakdowns to see whether the result differs by device or traffic source, and recordings to investigate where shoppers hesitate. An unexpected segment result can suggest another test, but a segment with few purchases gives you little evidence on its own. For what to test first, our Shopify CRO guide compares the changes that moved purchases on one store with those that didn’t.