Why Your App Shows Different Prices Than Your Friend: A Friendly Guide to A/B Testing

Two people, one app, two different experiences

Picture this: you and a coworker are ordering lunch from the same delivery app. You’re sitting side by side. Same restaurant. Same time. Yet your screen shows “Free delivery over $20” while theirs shows “$2.99 delivery fee”. Or your checkout button says “Place order” and theirs says “Pay now”. It’s easy to assume something is broken—or worse, that you’re being singled out.

Often, it’s neither. What you’re seeing is frequently a simple analytics practice called A/B testing: a controlled experiment where a company shows Version A to some people and Version B to others, then compares what happens.

A/B testing is everywhere: shopping sites, streaming services, banking apps, airline booking pages, fitness apps, even internal workplace tools. And it’s not just about making things “prettier.” It’s about answering practical questions like:

• Do people finish sign-up faster if we remove one form field?
• Does a different reminder message reduce missed appointments?
• Will more people read an important policy update if it’s shorter?

Think of it like a bakery offering two cookie samples and watching which tray empties first. The difference is: in digital products, the “cookie samples” might be button text, image choices, or how a price is displayed.

Newsletter

Never miss out on great articles — just enter your email and receive a weekly update!

Go

I have read and accept the Terms & Policy

What A/B testing really is (and why it works)

At its core, an A/B test is a fair comparison. You change one thing (or a small set of things) and keep everything else as similar as possible. Then you measure a behavior you care about—like clicking a button, finishing a purchase, or returning the next day.

Here’s a clean, everyday example:

• Question: Will more people book a dentist appointment if the button says “Book now” instead of “See times”?
• Version A: Button says “See times”
• Version B: Button says “Book now”
• Metric: Appointment bookings per 1,000 visitors

Users are typically assigned to A or B randomly. Random assignment is the secret sauce: it helps ensure the groups are similar overall. That way, if you see a difference in outcomes, there’s a good chance it’s due to the change—not because one group happened to include more weekend shoppers or more people on faster Wi‑Fi.

Companies like A/B tests because they’re concrete. Instead of debating preferences (“I like this headline better”), they can ask: Which version helped more people do what they came here to do?

Why changes can feel surprisingly powerful

In everyday life, tiny framing shifts can change what people choose. A restaurant menu that highlights one dish as “Most Popular” can steer orders. A friendly reminder can reduce no-shows. In apps, “framing” is often the difference between:

• “Start free trial” vs. “Try it free for 7 days”
• Showing the monthly price first vs. the yearly price first
• One long paragraph vs. three short bullet points

These aren’t mind tricks as much as clarity tests. Many A/B tests succeed simply by removing confusion.

What gets tested in real products? Common categories include:

That last category—pricing presentation—is a big reason people notice tests. Seeing different prices can happen for multiple reasons (region, taxes, timing, inventory, membership tier), but A/B testing is one common cause of different price displays and different discount messages—even when the underlying base price is the same.

A/B testing vs. personalization (they’re not the same)

It helps to separate two ideas that look identical on the surface:

In reality, companies often use both. A/B testing helps them decide what “default experience” is best. Personalization tweaks that default for different groups.

The hidden mechanics: how a “simple test” can go wrong

A/B testing sounds straightforward, but the details matter. Many confusing outcomes come from tests that are “almost right” but not quite.

1) The “peeking” problem

Imagine flipping a coin 10 times. At flip #3 you might be at 3 heads, 0 tails and feel confident the coin is “biased.” But you’d be fooling yourself. The same happens in A/B testing when teams check results too early and stop the test the moment one version is ahead. That’s called peeking, and it increases false wins.

Real-life analogy: it’s like declaring a soccer winner after 12 minutes because one team scored first.

2) Audience mixing

Say Version B launches on Monday, and on Monday a big marketing campaign also starts. Now Version B gets more “new visitors” than Version A, and new visitors behave differently. The test outcome can be driven by the audience shift, not the design change.

Good practice is to randomize properly and ensure both versions receive similar traffic sources and device types.

3) Too many changes at once

If Version B changes the headline, the layout, the images, and the price display, and performance improves, you still won’t know why. Sometimes that’s fine (you just want the best-performing page). But if you want learning you can reuse, smaller tests are easier to interpret.

4) Measuring the wrong thing

A classic trap is optimizing for a metric that’s easy to move but doesn’t reflect real value.

• Changing a button to bright red might increase clicks (good), but if it leads to more cancellations later (bad), the “win” is not really a win.
• A more aggressive pop-up might increase email sign-ups (good) while making people leave the site sooner (bad).

That’s why teams often track a primary metric (the main goal) and guardrail metrics (things that shouldn’t get worse), such as refunds, complaints, or time-to-complete.

5) Small numbers, big confidence

If only 40 people saw Version A and 42 people saw Version B, and B “wins,” that win may just be noise. Small samples naturally bounce around. The fix is usually boring but effective: let the test run longer or expose it to more people.

A practical “spot the test” checklist

If you’re curious whether you’re in an A/B test, here are clues that often show up in daily life:

• You see a slightly different button text, layout, or banner than someone nearby.
• The change feels cosmetic or wording-related rather than a whole new feature.
• The difference persists even after refreshing (many tests “stick” you to A or B for consistency).
• The app feels like it’s “trying out” a new message—especially during sign-up, checkout, or notifications.

None of these guarantees it’s an A/B test, but they’re common patterns.

Why companies don’t just “ask users” instead

Teams do run surveys and interviews, but people often say one thing and do another—without meaning to. You might tell a streaming app you want “more variety,” then spend the weekend watching the same comfort show. A/B tests measure behavior in context, not opinions in a vacuum.

Is it fair that my friend gets a better deal?

This is the emotional core of it. When tests touch pricing, fairness becomes personal. Companies may test:

• How to display discounts (percent off vs. dollars off)
• Whether to show shipping costs early or at checkout
• Whether a coupon banner increases completed purchases

Sometimes the underlying final price is identical; only the framing differs. Sometimes the final price can differ because the test is intentionally experimenting with offers. Whether that’s “fair” depends on the context, laws, and company policy—but from an analytics standpoint, the goal is usually to learn what presentation or incentive changes behavior.

If you want to sanity-check what’s happening in your own life, try a few low-effort comparisons:

• Check the same product logged in vs. logged out.
• Compare mobile vs. desktop.
• Try a different browser profile (not to “hack” anything—just to see if the experience changes).

These won’t prove it’s an A/B test, but they can reveal whether the experience is stable or variable.

FAQ-style curiosities people often have