A/B testing,in plain English.

A test where both versions are identical, used to check the testing setup itself.

If two identical pages show a “significant” difference, something is wrong: the split, the tracking, or how the results are read. A/A tests are also a good way to see how often pure noise looks like a winner.

Two versions of a page, shown to random halves of your visitors, to see which sells more.

Visitors are split at random between the original (A) and a changed version (B). Because the split is random, the only systematic difference between the two groups is the change, so a difference in results can be put down to it. Also called a split test.

The share of visitors (or product page views) that add something to the cart.

A useful early signal for product page tests: it moves faster than orders, but a higher add-to-cart rate doesn’t always mean more orders. Read it next to conversion rate, not instead of it.

Revenue divided by the number of orders.

Some changes raise conversion but lower order value, or the other way round. That’s why revenue per visitor, which combines both, is often the better thing to judge a test on.

A way of reading test results as the probability that one version is better than another.

Instead of asking “how surprising is this data if nothing changed?” (a p-value), a Bayesian analysis answers the question people actually ask: “how likely is it that B is better than A?” It still needs enough data, and it still shouldn’t be read over and over until it says what you hope.

The share of visits that leave after seeing a single page without doing anything.

A high bounce rate on a landing page often means the page doesn’t match what brought the visitor there, or it loads too slowly. On its own it says nothing about sales; use it to find pages worth a closer look.

When a shopper adds something to the cart and leaves without buying.

Research on why shoppers abandon points at the same few reasons again and again: unexpected costs like shipping, being made to create an account, slow delivery, not trusting the site with their card, and a checkout that’s too long.

The probability that a variant is better than the original, given the data so far.

“B has a 96% chance of beating A” means that, given everything observed, there’s a 96% probability that B’s true conversion rate (or revenue per visitor) is higher. It doesn’t say how much higher; look at the range of likely lifts for that.

The original version in a test, the one everything else is compared with.

Usually version A. A good test changes one thing against the control, so you know what made the difference.

The share of visitors (or sessions) that complete a goal, usually an order.

Orders divided by visitors or sessions. Which one matters: per-session rates look lower than per-visitor rates for the same store, because one visitor can come back several times before buying.

Finding out why visitors don’t buy, and testing changes that fix it.

Good CRO is research plus experiments: look at where and why shoppers drop out, form a hypothesis about a fix, and test it against the original. Changing things and hoping isn’t CRO; it’s redesigning.

The range a true value most likely sits in, given the data. The Bayesian cousin of a confidence interval.

A 90% credible interval for lift of +2% to +11% means there’s a 90% probability the true lift is somewhere in that range. A narrow interval means you know a lot; a wide one, not much yet.

Signs that a visitor is about to leave, like the cursor heading for the tab bar.

On a computer it’s usually the cursor moving fast towards the top of the window; on a phone, a quick scroll back up. It’s the moment for one polite question, not a wall of discount popups.

Missing a real improvement because the test couldn’t see it.

Usually the result of a test that’s too small: not enough visitors or orders to tell a real but modest lift from noise. Planning the sample size up front is how you avoid it.

Declaring a winner that isn’t one.

Pure chance makes one version look better from time to time. Checking results over and over and stopping at the first good-looking moment makes false winners far more common than the dashboard admits.

When visitors briefly see the original page before the test version replaces it.

Also called flash of original content. Besides looking broken, it can bias the test: visitors who saw the original flash aren’t really seeing the variant. Good testing tools apply changes before the page paints.

A slice of visitors kept away from all changes, to measure their combined effect.

Individual tests each measure one change. A holdback measures everything at once: a few percent of visitors keep the old experience, and the difference in revenue per visitor between them and everyone else, over the same period, is what all the changes together earned.

A testable guess: if we change X, Y will improve, because Z.

The “because” matters. A test without a reason teaches you nothing when it loses. A good hypothesis comes from evidence: research, analytics, or what shoppers told you.

How much better (or worse) a variant performed than the control, in percent.

If the control converts at 2.0% and the variant at 2.2%, the lift is +10%. Always read a lift with its range: “+10%” from a tiny test can easily be anything from −5% to +25%.

The smallest lift a test is set up to reliably detect.

The smaller the effect you want to catch, the more visitors you need, and the need grows fast: detecting a lift half as big needs roughly four times the traffic. On a quiet store, test bold changes, not button colours.

Testing several changes at once in every combination.

It tells you which combination works best and how changes interact, but it needs much more traffic than an A/B test, because traffic is split across every combination. Most stores are better off with A/B tests.

A temporary bump because something is new, not because it’s better.

Returning visitors may click a changed element just because it’s different. Running tests for full weeks, and long enough for the novelty to wear off, keeps it from fooling you.

How surprising the data would be if there were really no difference.

A p-value of 0.03 means: if A and B were truly the same, you’d see a difference at least this big about 3% of the time. It is not the probability that B is better, which is the most common misreading.

Checking test results repeatedly and stopping as soon as they look significant.

Every look is another chance for noise to cross the line. Stop at the first good-looking moment and you’ll crown far more fake winners than your stated error rate. The fix is a planned sample size, read once.

Showing different visitors different versions of a page, on purpose.

For example by the ad they came from, their device, or whether they’ve been before. It should still be tested: a personalized page needs a holdout that sees the usual page, or you can’t know it helps.

Total revenue divided by the number of visitors.

It combines conversion rate and order value in one number, so it catches changes that sell more often but smaller (or less often but bigger). It’s noisier than conversion rate, so it needs more data.

When a 50/50 test doesn’t end up close to 50/50.

Random splits are never exactly even, but a big gap (say 52/48 on thousands of visitors) usually means something is broken: a redirect, a bot filter, or a variant that crashes for some visitors. Results from a test with SRM can’t be trusted.

How many visitors (and conversions) a test needs before it can be read.

It depends on your current conversion rate, the smallest lift you care about, and how sure you want to be. Work it out before the test starts, then stick to it.

A subgroup of visitors, like phone users or visitors from one campaign.

Results can differ by segment: a change can win on phones and do nothing on desktop. But slice a test into enough segments and one will “win” by chance, so treat segment findings as ideas for the next test.

An A/B test where each version lives at its own address.

Useful for testing whole new pages or flows. Visitors are redirected to one of the addresses at random, which makes flicker and tracking a bit harder to get right.

The chance that a test detects a real effect of a given size.

80% power means that if the lift you planned for is real, the test finds it 4 times out of 5. Low power is why small tests so often end with “no difference”.

A result unlikely enough under “no difference” to take seriously.

Traditionally p < 0.05. It’s a property of a planned test, read once; it loses its meaning if you keep checking and stop when it appears.

How visitors are divided between versions, usually evenly.

An even split gets you to a result fastest. Sending most traffic to the original “to be safe” slows the test down a lot; protecting revenue is better done with guardrails that stop clear losers.

Tags on a link that say where a visitor came from: utm_source, utm_medium, utm_campaign.

Ad platforms add them to the links in your ads. They’re how analytics, and personalization, know that a visitor came from a particular campaign.

A changed version of a page in a test, as opposed to the control.

Version B (and C, D…). A good variant changes one idea clearly enough that visitors would notice.