How to Set Up a PostHog A/B Test on Shopify

You can run A/B tests on Shopify with PostHog, but the setup needs to be handled carefully.

PostHog experiments are powered by feature flags. Shopify themes are powered by templates, sections, blocks, app embeds, and custom scripts. To make the two work together, you need a clean way to show the right Shopify element to the right visitor, then send the result back into PostHog so the experiment can measure conversion impact.

This guide walks through the practical setup: duplicate the Shopify theme, create the PostHog experiment, prepare stable selectors, add the variant script, QA every variant, publish the theme, and launch the experiment with a primary metric.

The goal is not just to swap a headline or section. The goal is to run a Shopify experiment that produces data you can trust.

What You Need Before You Start

Before building the experiment, make sure you have:

• A working PostHog account.
• PostHog installed on the Shopify storefront.
• Admin access to Shopify.
• Permission to duplicate and publish themes.
• A clear hypothesis for the test.
• A page, template, section, or element you want to test.
• Stable CSS IDs or data attributes for the control and variant elements.
• A conversion event or funnel in PostHog.

If PostHog is not installed on Shopify yet, do that first. The script in this guide assumes window.posthog is already available on the storefront.

Step 1: Define the Test Before Touching the Theme

Start with the decision you want the experiment to support.

Examples:

• Does a new product-page hero section increase add-to-cart rate?
• Does a shorter above-the-fold offer increase checkout starts?
• Does a different product benefit block increase purchases?
• Does a new CTA section improve lead or form submissions?

Write the test as a simple hypothesis:

Changing [element] from [control] to [variant] will improve [metric] because [reason].

Example:

Changing the product-page hero headline from product-focused copy to outcome-focused copy will improve add-to-cart rate because visitors will understand the value faster.

Keep the first version narrow. If you change the headline, section order, image, price framing, and CTA at the same time, PostHog may show a winner, but you will not know which change caused the result.

Step 2: Duplicate the Shopify Theme

Do not build the test directly on the live theme unless the change is tiny and you have a rollback plan.

In Shopify admin, go to:

Sales channels -> Online Store -> Themes

Then:

1. Find the current live theme.
2. Click Actions.
3. Click Duplicate.
4. Rename the duplicated theme, for example Test Theme - Product Hero Experiment.
5. Open Customize on the duplicated theme.

The duplicate theme gives you a safe place to build and QA the experiment. It also gives you a quick rollback path if something breaks after launch.

Keep the original live theme unchanged until QA is complete.

Step 3: Create the Experiment in PostHog

In PostHog:

1. Open Experiments.
2. Click New experiment.
3. Name the experiment.
4. Create the feature flag key.
5. Add the variants.
6. Save the experiment as a draft.

Use a naming format that stays readable later:

[Experiment number] - [Page or funnel step] - [Test hypothesis]

Example:

001 - Product page - Hero headline test

Use a code-friendly feature flag key:

001-product-page-hero-headline-test

Use simple variant keys:

control
test_group_1
test_group_2

The feature flag key is important because your Shopify script must reference it exactly. A mismatch between the PostHog key and the script key is one of the easiest ways to break the test.

Step 4: Prepare the Shopify Variant Elements

In the duplicated Shopify theme, open the template or page you want to test.

You need one targetable element for the control and one targetable element for each variant.

For example:

• Control hero section.
• Test group 1 hero section.
• Test group 2 hero section.

If you are testing a section, duplicate the section and edit only the part being tested. If you are testing copy, keep the layout and styling the same. When you are testing layout, keep the copy as consistent as possible.

The cleaner the difference, the cleaner the experiment readout.

Step 5: Add Stable CSS IDs or Data Attributes

Your script needs stable selectors so it can hide and show the correct version.

Good selectors:

id="experiment-control"
id="experiment-tg1"
id="experiment-tg2"

or:

data-experiment-variant="control"
data-experiment-variant="test_group_1"
data-experiment-variant="test_group_2"

Avoid generic selectors like:

.button
.hero
.heading
.section

Those can match multiple elements and make the test fragile.

If the theme editor does not let you add IDs directly, ask a developer to add IDs or data attributes in the Liquid section, block, or snippet that renders the element.

Document the selectors before writing the script:

control: #experiment-control
test_group_1: #experiment-tg1
test_group_2: #experiment-tg2

Step 6: Hide Test Variants by Default

Hide the test variants before the PostHog feature flag response is available.

For variant sections, add CSS like:

#experiment-tg1,
#experiment-tg2 {
display: none;
}

Leave the control visible as the safe fallback. The script can hide all versions after PostHog loads and then show the assigned variant.

This reduces flicker and prevents visitors from seeing multiple versions before the feature flag is ready.

Step 7: Add the A/B Test Script to Shopify

Add the experiment script in a place that loads on the tested page. The exact location depends on your theme.

Common options:

• A custom liquid section near the bottom of the tested template.
• A custom HTML or liquid block above the footer.
• A theme code file used only on the target template.

Use a page-specific location when possible. Do not scatter experiment scripts across unrelated templates.

Update the featureFlagKey and selectors before publishing:

PostHog experiments use feature flags, so the script waits for feature flags with onFeatureFlags() and then checks the assigned variant with getFeatureFlag().

If PostHog does not load, the script falls back to the control. That is intentional. A broken analytics request should not break the storefront.

Step 8: Hide the Script Section Itself

If you add the script through a visible Shopify custom content section, make the section invisible so it does not affect the page layout.

In the section or custom CSS area, use:

display: none;

Only hide the wrapper section that holds the code. Do not accidentally hide the actual variant sections unless that is part of the intended setup.

Step 9: QA Every Variant Before Launch

Preview the duplicated theme and test each variant visually.

Check:

• Control appears correctly.
• Test group 1 appears correctly.
• Test group 2 appears correctly, if used.
• Only one variant appears at a time.
• The layout works on mobile.
• The layout works on desktop.
• Product forms still work.
• Add-to-cart behavior still works.
• Checkout entry still works.
• PostHog events are still captured.

PostHog supports overriding feature flags from the browser console for testing. After PostHog has loaded, run:

posthog.featureFlags.overrideFeatureFlags({
flags: {
'001-product-page-hero-headline-test': 'test_group_1'
}
});
posthog.reloadFeatureFlags();

Then refresh and confirm the correct Shopify element appears.

Repeat for:

control
test_group_1
test_group_2

The original workflow used URL parameters like:

https://example.com/products/example-product?001-product-page-hero-headline-test=test_group_1

That can work if your site has custom logic that reads URL parameters and overrides the flag for QA. It is not the default PostHog override method by itself. If you want URL-based QA, add and document that override behavior intentionally.

Step 10: Set the Primary Metric in PostHog

Do not launch the experiment without a primary metric.

For Shopify, useful primary metrics include:

• Add to cart.
• Checkout started.
• Purchase.
• Form submitted.
• Product page to checkout funnel.
• Product page to purchase funnel.

In PostHog:

1. Open the draft experiment.
2. Scroll to Primary metrics.
3. Click Add primary metric.
4. Choose Funnels if the test should measure a conversion path.
5. Define the steps.
6. Set the attribution type.
7. Set the conversion window.
8. Save the metric.

Example funnel:

Feature flag called -> Product added to cart -> Purchase

For product-page tests, purchase is usually the strongest metric, but it may need more traffic. Add-to-cart or checkout started can be useful secondary metrics when purchase volume is low.

Avoid using pageviews as the main success metric for revenue tests. A pageview usually does not tell you whether the variant improved the business outcome.

Step 11: Publish the Shopify Theme

Once QA is complete, publish the edited theme.

In Shopify:

1. Go to Online Store -> Themes.
2. Find the duplicated test theme.
3. Click Actions.
4. Click Publish.
5. Confirm that it is now the current theme.

Immediately after publishing, run one more live-site check:

• Visit the public page.
• Confirm one variant appears.
• Confirm product actions work.
• Confirm PostHog Activity receives events.
• Confirm the feature flag is called.

Step 12: Launch the Experiment in PostHog

After the live Shopify theme is confirmed:

1. Open the experiment in PostHog.
2. Confirm the feature flag key.
3. Confirm the variants and traffic split.
4. Confirm the primary metric.
5. Click Launch.

After launch, watch the first visitors carefully. In PostHog, check that users are flowing into the variants and that the primary metric starts receiving data.

How to Verify the Experiment Is Working

Check the setup at three levels.

Visual QA

Force each variant and confirm the right Shopify section appears. Check mobile and desktop.

Event QA

In PostHog Activity, confirm that pageviews, product events, add-to-cart events, checkout events, and purchases are captured as expected.

Experiment QA

In the experiment results, confirm:

• Traffic is assigned to variants.
• The feature flag is being called.
• The primary metric increments.
• No variant has suspiciously low or zero traffic.
• The conversion path matches the actual Shopify flow.

If a variant gets traffic but no conversions after a meaningful period, check the selector, script, metric definition, and Shopify event tracking before assuming the variant is bad.

Common Mistakes to Avoid

Editing the live theme directly

Build in a duplicated theme first. It gives you a safer QA environment and a faster rollback path.

Using unstable selectors

Generic classes can match multiple Shopify elements. Use CSS IDs or data attributes that are unique to the experiment.

Hiding the wrong section

When hiding the script wrapper, make sure you are not hiding the visible test content by mistake.

Forgetting to wait for feature flags

On a first visit, feature flags may not be immediately available. Use onFeatureFlags() before deciding which variant to show.

Using a URL override without implementing it

A URL like ?feature_flag_key=test_group_1 only works if the storefront script reads that query parameter and applies the override. PostHog’s documented client-side override uses posthog.featureFlags.overrideFeatureFlags().

Choosing a weak primary metric

Use a metric close to revenue or the core funnel action. For Shopify, purchase, checkout started, and add to cart are usually stronger than pageview.

Running overlapping tests on the same element

Multiple experiments can run at the same time, but they should not compete over the same hero, product form, CTA, or checkout path unless you have a clear interaction plan.

Why This Matters for Shopify CRO

Shopify A/B testing only helps if the result is trustworthy.

A clean PostHog setup lets you see:

• Which variant each visitor saw.
• Whether the variant changed add-to-cart behavior.
• Whether checkout starts improved.
• Whether purchases improved.
• Whether the result differs by traffic source, device, or campaign.
• Whether session recordings show friction in one version.

That is the difference between changing a theme and running a real CRO experiment.

PostHog is especially useful when you want experiments, funnels, session replay, and event analysis in one place. Instead of only asking “which version won?”, you can inspect why it may have won and what to test next.

Frequently asked questions

External & Internal References

• How to install PostHog on Shopify
• How to set up a PostHog A/B test on WordPress
• PostHog Experiments documentation
• PostHog-Shopify docs
• Shopify theme duplication and publishing documentation