Why Your Control Group Could Be Ruining Your Incrementality Test

The Key to More Accurate Incrementality Testing in Digital Advertising

When running incrementality tests in digital advertising, the right control group can make or break your results. If your control group isn’t truly comparable to your test group, your incremental lift measurement could be wildly misleading — giving you inflated (or deflated) results that don’t actually reflect your campaign’s success.

Recently, Steven Ohrnstein, Viant’s SVP of Platform & Analytics, shared on LinkedIn how too many marketers fall into this trap. He explained that an improperly selected control group can make results “look too good to be true” — when in reality, it’s just bad methodology. His takeaway? Similarity matters. Without a comparable control group, you’re not measuring lift — you’re measuring noise.

What Is an Incrementality Test?

An incrementality test is a method used in digital marketing measurement to determine how much impact an ad campaign has beyond organic behavior. This is done by comparing a test group (exposed to the campaign) with a control group (not exposed). The difference in behavior between the two groups represents the incremental lift driven by the campaign.

But here’s the catch: if the control group isn’t properly selected, your results won’t be accurate.

Not All Control Groups Are Created Equal

Selecting the wrong control group is like comparing apples to oranges.

Example: The Regional Burger Chain Effect 🍔

Let’s say a southern California burger restaurant chain runs an ad campaign and measures incremental lift by comparing Southern California customers (test group) to a nationwide control group. The result? Massive lift—but it’s meaningless. Why? Because the southern California burger restaurant chain isn’t available in most states, meaning the control group never had the opportunity to convert in the first place.

This kind of flawed methodology exaggerates performance and leads marketers down the wrong path when optimizing ad spend.

The solution? Control groups must be as similar as possible to the test group — in demographics, geography and behavior—to get reliable incrementality insights.

3 Types of Control Groups in Digital Advertising (and Their Pros & Cons)

Not all incrementality measurement methods are created equal. Here are the three main types of control groups, along with their trade-offs:

1. Randomized Control Trials (RCTs) 🎯

✔ Gold standard for causal measurement — eliminates selection bias
✔ Randomly assigns people to test or control groups

❌ Risk: If the randomization isn’t carefully designed, the control group may still be too different from the test group (like the southern California burger restaurant chain example).

2. Public Service Announcement (PSA) Holdouts 🎗️

✔ Instead of suppressing ads, the control group sees a neutral PSA ad
✔ Reduces exposure bias since both groups still experience ad-serving

❌ Trade-Offs: This approach wastes media spend on non-promotional ads and doesn’t perfectly isolate the campaign’s impact.

3. Ghost Bidding 👻 (The New Standard)

✔ Uses lost ad auctions as the control group — ensuring identical conditions
✔ Same audience, same placements, same media dynamics — making it the most accurate method

❌ Requires demand-side platform (DSP) integration and technical expertise to implement effectively.

Why Your Control Group Must Mirror Your Test Group

Choosing the wrong control group can completely distort your incremental lift measurement. Here’s why:

✅ Accurate Attribution – If your control group isn’t exposed to the same purchase opportunities, lift results will be inflated.
✅ Reliable Insights – Demographics, location and behavior must closely match between test and control groups for valid data.
✅ Data-Driven Decision Making – A bad control group = bad data = wasted ad spend.

As Steven Ohrnstein emphasized in his LinkedIn post, marketers should scrutinize their control group methodology as much as they do their media strategy—because if your control group is too different from your test group, you’re not measuring lift—you’re measuring bad methodology. 🚀

Final Thoughts: How to Ensure a Strong Control Group

Marketers often focus on ad creative, targeting and budget, but the real key to measuring digital ad performance lies in choosing the right control group. Whether using RCTs, PSA holdouts, or ghost bidding, the goal is the same: keep the control group as close to the test group as possible to get accurate, actionable insights.

Want to learn more about incrementality testing, programmatic advertising and AI-driven marketing measurement? Explore Viant’s AI-powered ad solutions to optimize your campaign performance with precision.