BRUTAL MARKETING
ECOMMERCE A/B TESTING: 11 EXPERIMRNTS WITH REAL NUMBERS
2026
BRUTAL MARKETING
Ecommerce A/B Testing: 11 Experiments With Real Numbers
2026
A/B Testing for Ecommerce: 11 Experiments With Numbers and How to Avoid False Wins
Here's a story we see all the time. The team makes the “Buy” button high-contrast, site orders go up 14%, everyone celebrates. A month later the owner opens the revenue report, and it has barely moved. Some of the “extra” orders were never confirmed, some came back as returns, and some cash-on-delivery parcels were never picked up.
That's the most expensive A/B testing mistake in ecommerce: picking the winner by clicks on the site instead of money in the bank.
That's the most expensive A/B testing mistake in ecommerce: picking the winner by clicks on the site instead of money in the bank.
Serhii
Ponomarenko
Ponomarenko
At Brutal Marketing we run into it regularly when we connect an online store's CRM to its web analytics. The test “won” in Google Analytics and lost in the paid-orders report.
Below are 11 publicly documented ecommerce A/B experiments with specific numbers. You'll also see how to write a hypothesis, how much traffic an honest result needs, which mistakes distort conclusions, how to trace a test all the way to revenue through your CRM, what replaced Google Optimize, and what to do in your first 30 days.
Below are 11 publicly documented ecommerce A/B experiments with specific numbers. You'll also see how to write a hypothesis, how much traffic an honest result needs, which mistakes distort conclusions, how to trace a test all the way to revenue through your CRM, what replaced Google Optimize, and what to do in your first 30 days.
What an A/B test is (and why it's not about button colors)
An A/B test is a controlled experiment. Two versions of a page or element are shown at the same time to randomly split visitors. Version A is the original (control), version B changes exactly one thing.
The key word is “one.” If you changed the headline, the photo and the button at once and conversion went up, you have no idea what worked.
The logic is simple: same traffic, one change, enough data. The answer you get isn't “better or worse” but “how many percent better, and how likely it is that this isn't chance.”
The key word is “one.” If you changed the headline, the photo and the button at once and conversion went up, you have no idea what worked.
The logic is simple: same traffic, one change, enough data. The answer you get isn't “better or worse” but “how many percent better, and how likely it is that this isn't chance.”
Before your first real test, run an A/A test: two identical versions. If the tool reports a “significant” difference between them, either the traffic split or the analytics is broken, and you can't trust any result that follows.
If you don't yet track conversion at each funnel step, start there. The basics worth fixing before any testing are covered in our guide on how to increase website conversion rate.
If you don't yet track conversion at each funnel step, start there. The basics worth fixing before any testing are covered in our guide on how to increase website conversion rate.
The hypothesis: the step most teams skip
“Let's try a blue button instead of a green one” isn't a hypothesis. It's a guess.
A working hypothesis has four parts: “If we [change] for [segment], then [metric] will move by [N%], because [observation].” Example: “If we show shipping cost and delivery time under the ‘Add to cart’ button, the share of paid orders will grow by 10%, because 30% of live chat questions are about shipping.”
The last part matters most. It forces you to lean on data, not on a designer's taste.
A working hypothesis has four parts: “If we [change] for [segment], then [metric] will move by [N%], because [observation].” Example: “If we show shipping cost and delivery time under the ‘Add to cart’ button, the share of paid orders will grow by 10%, because 30% of live chat questions are about shipping.”
The last part matters most. It forces you to lean on data, not on a designer's taste.
Where to find hypotheses
- Your GA4 funnel: the step where the largest share of visitors drops off.
- Heatmaps and session recordings (Microsoft Clarity is free): where people click, where they get stuck, how far they scroll.
- Your CRM: reasons for order cancellations and refused deliveries.
- Chats, emails and calls with your team: what customers ask before buying (“How much is shipping?”, “Can I return it?”).
- On-site search: what people look for and don't find.
In our experience at Brutal Marketing, the strongest hypotheses for a website live in the CRM and in customer conversations, not in design mockups. If a third of cancellations say “shipping didn't work for me,” a test with shipping terms on the product page will almost certainly beat a new banner. The first step is making those reasons visible: a sales funnel in CRM with a closed list of loss reasons.
How to prioritize: the ICE model
You'll always have more ideas than traffic. Score each from 1 to 10 on three criteria, Impact, Confidence and Ease, and multiply.
Our rule: one active test per funnel. Teams that run 5–7 tests at once on connected pages can't tie any single change to revenue afterwards.
11 ecommerce A/B experiments: the numbers and what they mean
The numbers below come from publicly documented experiments at different stores. They aren't a forecast for your site. They're a source of hypotheses: the same test on a different audience can return zero or a loss.
Product page
1. “Add to cart” button color. Replacing a grey button that blended into the background with a saturated green lifted add-to-carts by 14%. What worked wasn't green itself but contrast against that site's palette, so copying the color is pointless. The rule: the buy button should be the most visible element in the decision zone, especially on mobile.
2. Delivery countdown. “Free shipping” on the product page created no urgency. A live “Order by 5 PM, get it tomorrow” timer added 8.6% to conversion. Urgency only works when it's honest and tied to your real dispatch schedule, not a fake “only 3 left.”
3. Shipping details under the order button. Shoppers abandoned carts when shipping cost appeared as a surprise at the last step. Showing cost and delivery time right under the button on the product page raised sales by 60.1%. That matches Baymard Institute data: the average documented cart abandonment rate is 70.22%, and among shoppers who weren't just browsing, 40% leave because extra costs are too high. Show shipping options, costs and your returns policy on the product page.
4. Expert opinion. Product pages listed specs with no voice of authority. A specialist's commentary lifted purchase volume by 69% and revenue by 99%. Expert input beats reviews where a wrong choice is costly: electronics, tools, baby products, health.
2. Delivery countdown. “Free shipping” on the product page created no urgency. A live “Order by 5 PM, get it tomorrow” timer added 8.6% to conversion. Urgency only works when it's honest and tied to your real dispatch schedule, not a fake “only 3 left.”
3. Shipping details under the order button. Shoppers abandoned carts when shipping cost appeared as a surprise at the last step. Showing cost and delivery time right under the button on the product page raised sales by 60.1%. That matches Baymard Institute data: the average documented cart abandonment rate is 70.22%, and among shoppers who weren't just browsing, 40% leave because extra costs are too high. Show shipping options, costs and your returns policy on the product page.
4. Expert opinion. Product pages listed specs with no voice of authority. A specialist's commentary lifted purchase volume by 69% and revenue by 99%. Expert input beats reviews where a wrong choice is costly: electronics, tools, baby products, health.
Cart and checkout
5. “Submit” vs “Continue to payment.” A checkout button labeled “Submit” didn't tell shoppers what came next. Renaming it “Continue to payment options” raised click-through to the next step by 87.5%. Walk through your checkout: every “Next,” “OK” or “Submit” is a ready test candidate.
6. A cart that asks twice. Shoppers removed items on impulse and never came back. A “Remove or save for later?” prompt raised sales by 4.4%. That looks modest, but for a store with 1,000 orders a month it's 44 extra purchases without a dollar of ad spend.
7. Long form vs short form (B2B). A short quote form seemed ideal, but it led to a second page with more fields, and that surprise put people off. One detailed form showed the whole process upfront and lifted completions by 88.7%. Transparency matters more than field count: if a second step is unavoidable, put everything on one page.
6. A cart that asks twice. Shoppers removed items on impulse and never came back. A “Remove or save for later?” prompt raised sales by 4.4%. That looks modest, but for a store with 1,000 orders a month it's 44 extra purchases without a dollar of ad spend.
7. Long form vs short form (B2B). A short quote form seemed ideal, but it led to a second page with more fields, and that surprise put people off. One detailed form showed the whole process upfront and lifted completions by 88.7%. Transparency matters more than field count: if a second step is unavoidable, put everything on one page.
Email capture and repeat sales
8. Gamified signup. A standard “Leave your email, get 15% off” form converted poorly. A “spin the wheel” discount from 5% to 20%, with the same capture mechanics, brought +48.9% clicks, +25% submissions and +12.7% overall conversion. But check the list quality in your CRM: do these subscribers buy, or just collect coupons?
9. Segmented emails. An offer based on purchase history (bought cat food, got cat accessories) earned 61% more clicks than a generic promo to the whole list. That requires a CRM with order history, not just a list of email addresses. The simplest starting point is RFM analysis for ecommerce.
9. Segmented emails. An offer based on purchase history (bought cat food, got cat accessories) earned 61% more clicks than a generic promo to the whole list. That requires a CRM with order history, not just a list of email addresses. The simplest starting point is RFM analysis for ecommerce.
Navigation and catalog
10. A smart 404 page. Discontinued product URLs showed a plain “page not found.” Replacing it with similar product recommendations increased first-time purchases, add-to-carts and time on site. The same goes for out-of-stock items: keep the page, show alternatives and a “Notify me” button.
11. Category filters: an unexpected loss. For a category with several hundred products, advanced filters looked like an obvious improvement. They cut clicks by 27%. Filters shifted attention from products to search settings and pushed part of the range below the fold. The takeaway isn't “remove filters” but “don't ship the logical fix without testing it.”
11. Category filters: an unexpected loss. For a category with several hundred products, advanced filters looked like an obvious improvement. They cut clicks by 27%. Filters shifted attention from products to search settings and pushed part of the range below the fold. The takeaway isn't “remove filters” but “don't ship the logical fix without testing it.”
How much traffic and time an honest result needs
The popular advice “1,000 visitors per variant and you can draw conclusions” is almost always wrong for an online store. At a 2% conversion rate that's 20 orders per variant, and a difference of 5 orders will look like “+25%” while being pure noise.
The sample you need depends on three things: your baseline conversion rate, the minimum effect you want to detect (MDE), and the standard thresholds of 95% significance and 80% power. Use the table as a guide.
The sample you need depends on three things: your baseline conversion rate, the minimum effect you want to detect (MDE), and the standard thresholds of 95% significance and 80% power. Use the table as a guide.
Visitors per variant; the effect is relative (2% → 2.4% is +20%); two-sided test, 95% significance, 80% power.
How to read it: a store converting at 2% wants to detect a 20% lift. It needs about 21,000 visitors per variant, or 42,000 for the test. With 30,000 sessions a month, the test runs about six weeks.
How to read it: a store converting at 2% wants to detect a 20% lift. It needs about 21,000 visitors per variant, or 42,000 for the test. With 30,000 sessions a month, the test runs about six weeks.
Duration rules
- At least two full weeks. People buy differently on Monday and on Saturday, so a test must cover whole weekly cycles.
- No longer than 6–8 weeks. After that, cookies get cleared, people switch devices, and your clean groups fill with mixed traffic.
- Fix the sample size and end date before launch. Any sample size calculator will do: the one built into your testing tool or a standalone one like Evan Miller's.
What to do with low traffic
Below 10–15K sessions a month, classic A/B tests on purchase conversion take quarters. Workable alternatives:
The honest answer: sometimes you don't need an A/B test at all. If the cart doesn't show shipping cost, that's not a hypothesis to test. It's a bug to fix tomorrow.
- test bold changes, not shades: a restructured product page produces a big effect, and a big effect needs a smaller sample;
- measure a more frequent metric higher in the funnel, like add-to-cart instead of purchase;
- test in email and SMS, where the sample is your whole list and results arrive within a day;
- replace the test with qualitative research: session recordings, 5–7 customer interviews, reviews of support chats.
The honest answer: sometimes you don't need an A/B test at all. If the cart doesn't show shipping cost, that's not a hypothesis to test. It's a bug to fix tomorrow.
xx
1. Peeking and stopping early
Every company runs two processes: the written one (policies, instructions, flowcharts) and the live one (how work really happens). They never fully match.
We worked with a distribution company where policy said the rep qualifies the client, then sends a quote, then updates the status. In practice, 70% of deals started with an inbound call from a client who already knew everything and only wanted a price. Reps skipped qualification, not out of laziness, but because for that client type it was meaningless. The CRM was built around the policy, and it did not work.
After the research we rebuilt the pipeline around reality: two separate routes instead of one. Six weeks later, card completion went from 40% to 87%. Nobody wrote a new policy. The sales funnel in the CRM simply started matching how people actually sell.
We worked with a distribution company where policy said the rep qualifies the client, then sends a quote, then updates the status. In practice, 70% of deals started with an inbound call from a client who already knew everything and only wanted a price. Reps skipped qualification, not out of laziness, but because for that client type it was meaningless. The CRM was built around the policy, and it did not work.
After the research we rebuilt the pipeline around reality: two separate routes instead of one. Six weeks later, card completion went from 40% to 87%. Nobody wrote a new policy. The sales funnel in the CRM simply started matching how people actually sell.
2. Testing on atypical traffic
A test during a sale, Black Friday or an aggressive ad push shows how “hot” shoppers behave, not regular ones. November's winner may lose in March. The same applies to abnormal days: site outages, carrier delays, the launch of a new channel.
Run tests in stable periods and log unusual days in the test journal. With end-to-end sales analytics in place, you can see which sources currently send typical traffic and which are noisy.
Run tests in stable periods and log unusual days in the test journal. With end-to-end sales analytics in place, you can see which sources currently send typical traffic and which are noisy.
3. Not checking the traffic split (SRM)
You set a 50/50 split and got 50,000 visitors in A and 46,000 in B. That's not a detail, it's a sign something is broken: a redirect drops people, the variant script loads slower, bot filtering works unevenly. This is called Sample Ratio Mismatch, and with it you can't trust the result, even a “significant” one.
4. Ignoring the novelty effect
Returning customers click a new element simply because it's new. Week one shows +15%, three weeks later the effect is gone. Look at new and returning visitors separately: if only returning visitors drive the lift and it shrinks every day, that's novelty, not improvement.
5. Hunting for winners in segments after the test
“Overall there's no difference, but on iPhone in Texas variant B is up 40%.” Slice the data into 20 segments and at least one will “win” by chance. Write the segments you care about into the hypothesis before launch, and treat post-test findings as hypotheses for the next test, not as results.
And the sixth mistake, the most expensive one for an owner, is judging a test only by a website metric. It gets its own section.
And the sixth mistake, the most expensive one for an owner, is judging a test only by a website metric. It gets its own section.
Validating in your CRM: why a site winner can lose at the till
Google Analytics sees a placed order. But in ecommerce, a placed order isn't money yet. Some orders are never confirmed or get flagged as fraud, some cash-on-delivery parcels are refused at the door, some come back as returns. A test that won on clicks can lose on revenue.
A worked example
Variant A brought 1,000 orders, and 88% were kept: 880 paid orders. Variant B brought 1,100 orders (+10% in the website report), but only 78% were kept: 858 paid orders. B loses by 22 paid orders, and the store also pays reverse logistics on 242 returned or refused parcels versus 120 for A.
In Google Analytics, B is the clear winner. In the CRM, it's a clear loss. That's why at Brutal Marketing we insist on one rule: decide on a test only after the return and refusal data is in the CRM.
In Google Analytics, B is the clear winner. In the CRM, it's a clear loss. That's why at Brutal Marketing we insist on one rule: decide on a test only after the return and refusal data is in the CRM.
How to connect an A/B test to your CRM in 5 steps
- Your testing tool writes the variant ID (A or B) to a cookie or the dataLayer.
- At checkout, pass that ID in a hidden field along with the order, the same way you pass UTM parameters.
- Create an “A/B variant” field on the order or deal card in your CRM. In Kommo, Pipedrive or KeyCRM this is a custom field; the logic is the same everywhere.
- Build a report by variant in the CRM or on a business owner dashboard: confirmed, kept, cancelled with reasons, margin, repeat purchases.
- Make the call once most returns and refused deliveries are in, usually 2–4 weeks after the test ends, depending on your return window.
That's a closed loop: from a click on the site to money in the bank and a repeat order. We cover how it works for online stores in detail in our article on CRM for an online store and closed-loop sales.
Tests and customer segments
The same variant performs differently for different audiences. A delivery countdown may lift conversion for new visitors and change nothing for regulars who already know your terms. With a CRM, you can test on specific segments instead of the “average visitor”: new vs returning, one-time vs loyal, active vs lapsed. How to split your base for experiments like these is covered in our guide to customer segmentation.
What to test first
Not all tests are equal. Footer color and shipping cost on the product page compete for the same traffic, but their potential impact differs by an order of magnitude.
Analyze mobile and desktop separately: the same element sits in a different place on a phone screen. Alongside testing, set up automated customer communication in your CRM, including abandoned cart reminders that bring part of those orders back.
If you're just starting, don't try to cover everything. One well-planned test a month, with a clear hypothesis and enough data, beats ten chaotic ones.
If you're just starting, don't try to cover everything. One well-planned test a month, with a clear hypothesis and enough data, beats ten chaotic ones.
A/B testing tools after Google Optimize
Google Optimize is gone. Google shut it down years ago, yet it still appears in old guides as “the free way to start.” There's no free replacement with the same marketer-friendly visual editor: you either pay for convenience or bring in a developer.
How to choose. A small store without a developer should start with email tests, Clarity for hypotheses and a free trial of one visual tool for a specific test. With a developer, PostHog or GrowthBook cover most needs without a subscription.
But the tool isn't the point. An expensive platform with weak hypotheses delivers weak results, just faster. And none of them knows whether the customer kept the order. That data lives in your CRM.
But the tool isn't the point. An expensive platform with weak hypotheses delivers weak results, just faster. And none of them knows whether the customer kept the order. That data lives in your CRM.
Testing as a process: your first 30 days
The most common mistake is treating A/B testing as a one-off project: “We tested the button, found the better version, done.” Customer behavior, the product range and traffic all change, so what worked six months ago may perform differently today.
Launch plan
- Week 1: audit. Combine your GA4 funnel with CRM data: cancellations with reasons, returns, refused deliveries. Collect 10–15 hypotheses and score them with ICE. Fix obvious bugs, like shipping cost hidden until the last step, without testing.
- Week 2: setup. Pass the variant ID into the CRM, run an A/A test, calculate the sample size and end date for hypothesis #1.
- Weeks 3–6: first test. Launch it and don't peek. In parallel, run an email A/B test: a personal offer based on purchase history for your most active 20% of customers vs a generic promo.
- After the test: the decision. Wait for return and refusal data, compare variants by paid orders and margin, and record the result in your test log.
Test log
A log stops you from repeating old tests and makes failed ones useful too. A minimal template:
On-site A/B tests are only part of the system. A customer you keep working with after the first order brings in more than any optimized product page. How to build that part is covered in our article on customer retention in ecommerce.
We at Brutal Marketing will select the best CRM program for you to use in your business. We will be happy to tell you about the program's capabilities and show you which settings will exactly help you achieve the desired financial results.
Frequently Asked Questions
What is A/B testing for an online store?
It's a method where site traffic is randomly split between two versions of a page or element: the control (original) and a variant with one change. The goal is to find out which version brings more purchases, add-to-carts or form submissions.
How long should an A/B test run?
At least two full weeks and no longer than 6–8 weeks. The exact duration depends on the sample you need: at a 2% conversion rate, detecting a 20% lift takes about 21,000 visitors per variant.
How much traffic do I need to run A/B tests?
For tests on purchase conversion, roughly 30,000 sessions a month or more. With less traffic, test bold changes, measure more frequent metrics like add-to-cart, or test in email, where the sample is your whole list.
Can I test several elements at once?
A classic A/B test changes one element; otherwise you can't tell what drove the result. For several variables there's multivariate testing, but it needs many times more traffic and mostly suits large stores.
What replaced Google Optimize?
For marketers without a developer, VWO or AB Tasty with a visual editor. For teams with a developer, PostHog or GrowthBook, both with free tiers. For email, the built-in A/B testing in Klaviyo, Mailchimp, Omnisend and similar platforms.
Does A/B testing hurt SEO?
No, if you follow the basics: show search crawlers the same content users see, add rel=canonical pointing to the original in split URL tests, use temporary 302 redirects, and don't keep a test running longer than needed.
Why do I need a CRM if I have Google Analytics?
Google Analytics sees a placed order. The CRM sees whether it was confirmed, kept and paid for, how much margin it made, and whether the customer came back.
Get an ecommerce funnel audit: from click to kept order
Most stores optimize what they see in Google Analytics and miss where the money actually leaks: unconfirmed orders, returns, refused deliveries, one-time buyers.
At Brutal Marketing we review the whole funnel, from the first visit to payment and the repeat order. You get a list of A/B test hypotheses prioritized by impact on revenue, not clicks. If needed, we set up passing test variants into your CRM and a dashboard that shows results in paid orders and margin.
Request a funnel audit and CRM implementation → use the form below.
Want to check your readiness first? Go through the end-to-end analytics implementation checklist.
At Brutal Marketing we review the whole funnel, from the first visit to payment and the repeat order. You get a list of A/B test hypotheses prioritized by impact on revenue, not clicks. If needed, we set up passing test variants into your CRM and a dashboard that shows results in paid orders and margin.
Request a funnel audit and CRM implementation → use the form below.
Want to check your readiness first? Go through the end-to-end analytics implementation checklist.
By submitting an application, you agree to the privacy policy
© 2017-2027, Brutal Marketing
All rights reserved
© 2017-2027, Brutal Marketing
All rights reserved