Subscription offer test
A/B results
Compare the control and new versions. Everything is calculated in this browser; nothing you add here is uploaded or saved.
1. Orders
In Shopify admin: Orders → filter the test period → Export → "Plain CSV file". Then add the file here.
No file yet. You can also type the counts below.
2. Visitors
In Microsoft Clarity: filter by custom tag nl_ab_sub_offer = control, then new, for the same period. Use the same measure (users or sessions) for both.
Edit the order counts by hand
Filled in from the CSV. Change them only if you aren't using a file. Without a file, the AOV and RPV p-values can't be computed, because they need each order's value.
3. Results
| Metric | Control | New | Lift | p-value | Chance new is better | Power | Visitors needed (control / new) |
|---|
- CVR = orders ÷ visitors · AOV = revenue ÷ orders · RPV = revenue ÷ visitors · subscriber rate = subscription orders ÷ orders.
- p-value: two-sided, below 0.05 is the usual bar. Power: the chance this sample size detects the lift seen so far (aim for 80%). Visitors needed: per version, for 80% power at that lift and the expected split.
- An order belongs to a version when it carries the tag
ab-sub-neworab-sub-control. Subscription renewals aren't tagged, so they're never counted.
