A/B testing
An experiment, set up the way it would run in production
One live test on the product page: does showing the price on the buy button change add-to-cart? Assignment, display, measurement and the decision rules are all real. This store has no traffic, so there is no winner to report.
The test plan
- Hypothesis
- Showing the price on the button removes a reason to hesitate, so more visitors who see it add to cart.
- Control
- “Add to cart”
- Variant
- “Add to cart · $119.00”, on the main button of every product page.
- Primary metric
- Visitors who add to cart, divided by visitors exposed to the button.
- Guardrail
- Checkout starts per exposed visitor must not fall.
- Audience
- US market, split 50/50. Germany is excluded because assigning a visitor stores an identifier, and that needs consent first.
See both versions
These links set your group and mark your browser as a preview, so your visits stay out of the data.
This browser is in: not assigned (outside the test market)
How it works
- 01
Assigned at the edge
On the first page request the proxy picks a group at random and sets a first-party cookie, so a visitor stays in one group across pages and visits.
- 02
No flicker, still cached
Both versions are in the static HTML. A one-line script copies the cookie onto the page before first paint and CSS shows the right one, so the page never changes in front of the visitor and stays cacheable.
- 03
Counted when seen
Exposure fires only when the button is actually on screen and purchasable. Visitors who never reach it don't dilute the result.
- 04
Tied to orders
The group is written to the cart as a private attribute when the button is used, so it travels with the order. In GA4 it is a user property on every later event.
Decided before starting
- Run until the planned sample size is reached and for at least 14 days, to cover two weekly cycles.
- No early stopping on a good-looking day. Repeatedly checking inflates false positives.
- If the split differs from 50/50 beyond chance (sample ratio mismatch, p < 0.01), discard the data and find the cause.
- Ship the variant only if the primary metric wins at p < 0.05 and the guardrail holds.
How long would it need to run?
Two-sided alpha 0.05, 80% power, minimum 14 days.
- Visitors per variant
- 53,211
- Run time
- 54 days
Reading a result
Pre-filled with example numbers. They are not results from this store.
- Observed lift
- +15.0%
- p-value
- 0.072
Not significant. This doesn't show the versions are equal, only that the test can't separate them.
What this does and doesn't show
This proves the implementation and the method. It can't prove a conversion lift, because a concept store has no real shoppers. Events reach GA4 once a measurement ID is configured, and until then they appear in the event log of the “How it's built” panel.