How CDPs Support Incrementality Testing for Growth Teams

Blog

7/23/26

How CDPs Support Incrementality Testing For Growth Teams

Your paid media dashboard says the campaign generated $8 million in revenue.

That does not mean the campaign caused $8 million in revenue.

Some customers would have converted anyway. They may have already been searching for the brand, planning a purchase, responding to another channel, or returning because of an existing relationship.

Attribution assigns credit to marketing interactions that occurred before a conversion.

Incrementality testing asks a different question:

What would have happened if the marketing had not occurred?

That distinction can produce dramatically different conclusions.

In large field experiments at eBay, researchers found that the estimated returns from paid search were substantially lower than conventional nonexperimental estimates. Brand keyword advertising produced no measurable short-term benefit in that particular experiment. But later research across thousands of brands showed that the conclusion did not generalize universally: competitors can capture branded-search traffic when the focal brand stops bidding, and defensive brand advertising can create incremental value in those situations.

The correct conclusion is therefore not that brand search, retargeting, or another channel is inherently wasteful.

It is that your organization has to test its own incrementality.

Incrementality testing estimates the causal effect of marketing by comparing outcomes for a group that could receive the marketing treatment with a comparable group that could not. The difference between those outcomes is the incremental lift. It answers the question attribution cannot: would those conversions have happened anyway?

A customer data platform can strengthen these experiments by supplying eligibility rules, customer status, offline outcomes, owned-channel assignment, and persistent exposure records.

But a CDP does not automatically create a valid paid media holdout.

That distinction is critical.

Choose The Design: Five Ways To Run An Incrementality Test

The first decision is not which CDP segment to build.

It is which experimental design can actually answer the question.

1. Platform-Run Randomized Conversion Lift

Platforms such as Google can randomly split eligible users into treatment and control groups inside the advertising system itself.

The platform then withholds the campaign from the control group and compares conversions.

Best For

This is often the strongest practical choice when the question concerns one advertising platform.

It avoids a major problem with advertiser-uploaded exclusion lists because the advertising platform itself controls assignment and ad delivery.

Google currently supports user-based Conversion Lift across eligible Search, Display, Video, Demand Gen, App, and Performance Max campaigns. Google states that the study requires at least 1,000 observed conversions and a minimum $5,000 campaign budget.

Where CDP Data Helps

The CDP can supply:

The platform still performs the advertising randomization.

Limit

The result applies primarily to that platform and relies on its experimental and reporting infrastructure.

2. Ghost-Ad And Intent-To-Treat Designs

Ghost-ad designs create a more efficient counterfactual by identifying control users who would have been served an advertisement without actually buying a placebo impression for them.

This reduces the difference between people who had a genuine chance of ad exposure and people who did not.

The concept is particularly useful because simple intent-to-treat designs can become diluted when many people assigned to treatment are never actually exposed.

Best For

These approaches fit environments where the advertising platform or demand-side platform supports experimental control over potential exposure.

Where CDP Data Helps

The CDP can define eligible customer groups and join downstream outcomes.

It does not create the ghost-ad mechanism itself.

3. Geo Experiments

Geo incrementality changes advertising exposure or spend across comparable geographic regions and measures the difference in business outcomes.

Google currently supports geo-based Conversion Lift using comparable geographic groups, including holdback and go-dark study designs.

Best For

Geo experiments are particularly useful when:

  • User-level randomization is unavailable
  • The question spans multiple paid media channels
  • Offline store outcomes matter
  • Privacy or platform constraints prevent person-level experimentation

Where CDP Data Helps

The CDP can provide regional outcomes such as:

  • In-store transactions
  • CRM conversions
  • New versus existing customer revenue
  • Loyalty behavior
  • Repeat purchases

Limit

Geographies are noisier experimental units than individual customers.

Regional spillover, local promotions, seasonality, and uneven market composition can affect the result, so geo experiments often need more time and careful design.

4. Advertiser-Run Holdouts Using Exclusion Lists

This is the design that appears most naturally connected to a CDP.

The CDP randomly assigns customers to a treatment population and a holdout, then uploads the holdout as an exclusion audience.

For email, SMS, push, and on-site personalization, this can be powerful because the company controls delivery directly.

For paid media, the design is much weaker.

Best For

Use CDP-defined holdouts primarily in channels where the enterprise controls whether the treatment is delivered.

Limit In Paid Media

An ad platform can exclude only the people it successfully matches.

Everyone else in the intended holdout can remain eligible for advertising.

That means the actual control group is contaminated.

5. On-Off Or Pre-Post Testing

The simplest design turns spend off or changes spend during one period and compares the outcome with another.

Best For

This can provide directional evidence when budgets are too small for stronger experimental methods.

Limit

Seasonality, promotions, competitor activity, macroeconomic conditions, and changes in customer mix can all create differences unrelated to the advertising.

Use this approach when stronger designs are unavailable, and describe the causal confidence accordingly.

Why CDP Holdouts Leak In Paid Media

A CDP may perfectly assign 100,000 customers to a holdout.

That does not mean the advertising platform will suppress all 100,000.

Several layers create leakage.

The Platform Must Match The Customer

Customer-list platforms match uploaded identifiers to accounts they recognize.

The number uploaded and the number available for advertising are therefore different.

This alone means a CDP-defined control group does not equal a platform-enforced control group.

The List Takes Time To Process

Google's current Customer Match API documentation says an upload job can complete in as little as 10 minutes but can require up to 24 hours.

If the campaign begins before the exclusion is fully processed, holdout customers can still receive advertising during the early part of the test.

Consent Can Reduce The Uploadable Population

For EEA users, Google states that customer data without the necessary consent is treated as unconsented and cannot be processed for Customer Match ad personalization.

Those customers may therefore exist in the CDP holdout while remaining absent from the advertising-platform exclusion.

Identity Errors Can Put One Person In Both Arms

A false split can create two profiles for one person.

One profile may be assigned to treatment while the other enters the holdout.

A false merge can create the opposite problem by incorrectly combining different people.

Identity quality is therefore part of experimental validity whenever the CDP controls assignment or joins outcomes.

Contamination Pushes Lift Toward Zero

When members of the supposed control group continue receiving advertising, treatment and control become more similar.

The measured difference shrinks.

That means an exclusion-list experiment can report little or no lift even when the campaign has a real effect.

For paid media, treat a leaky exclusion-list result as a lower-bound estimate, not automatically as proof that the campaign created no incremental value.

Measure The Leakage

If an exclusion-list holdout is unavoidable, calculate:

Coverage Ratio = Platform Matched Holdout Audience ÷ CDP Holdout Audience

Also record:

The lower the coverage, the more cautious the interpretation should become.

For important paid media budget decisions, prefer platform-run randomization or a well-designed geo experiment when available.

Size The Incrementality Test Before You Run It

A poorly powered experiment can produce an accurate answer to the wrong question:

“We did not have enough data to detect the effect.”

That is not the same as:

“The advertising had no effect.”

Lewis and Rao analyzed 25 large advertising field experiments and found that sales variability makes advertising effects unusually difficult to measure. Even experiments involving millions of customers can produce wide confidence intervals around ROI.

Google addresses the same practical problem through its Conversion Lift study-power estimate. Its interface currently expresses study certainty between 50% and 95%, and Google recommends targeting at least 90% for stronger conclusions.

Illustrative Sample Size: 5% Baseline And 20% Relative Lift

Suppose baseline conversion is 5%.

You want to detect an increase to 6%, which is a 20% relative lift.

Under a simple two-arm user-level experiment using approximately 95% confidence and 80% power, you would need roughly:

  • 8,200 users per arm
  • 16,300 users total

Illustrative Sample Size: 5% Baseline And 10% Relative Lift

Now suppose you need to detect movement from 5% to 5.5%.

The requirement rises to approximately:

  • 31,200 users per arm
  • 62,500 users total

Illustrative Sample Size: 5% Baseline And 5% Relative Lift

Detecting movement from 5% to only 5.25% requires roughly:

  • 122,000 users per arm
  • 244,000 users total

Illustrative Sample Size: 2% Baseline And 10% Relative Lift

At a 2% baseline, detecting movement to 2.2% requires approximately:

  • 80,700 users per arm
  • 161,000 users total

These are planning illustrations, not universal sample-size requirements.

The actual calculation should account for the real baseline, minimum detectable effect, exposure probability, clustering, experimental unit, and design.

The practical lesson is more important than the exact numbers:

Halving the effect you want to detect can require roughly four times as much sample.

Duration Matters Too

The experiment needs enough time to capture the conversion lag.

Google currently permits user-based Conversion Lift studies as short as seven days but generally recommends more than 14 days, especially when conversions occur slowly.

A seven-day test for restaurant ordering is fundamentally different from a seven-day test for enterprise software, travel, automotive, or another long-consideration purchase.

Define duration around customer behavior, not around the reporting calendar.

What Should Growth Teams Test First?

The best first incrementality test is usually a material spend category where the organization is uncertain how much behavior the media actually changes.

Brand Search

Brand search is an obvious candidate because many people searching for the company are already demonstrating high intent.

The eBay field experiments found no measurable short-term benefit from brand keyword advertising in that environment. But research across thousands of other brands later showed that competitors can capture meaningful traffic when a focal brand disappears from the sponsored results.

Conclusion: Test brand search. Do not assume either that it is wasted or that it is indispensable.

Retargeting

Retargeting intentionally focuses spend on customers who already exhibited purchase intent.

That makes attribution particularly vulnerable to confusing existing intent with advertising effect.

Run platform lift, ghost-ad, or geo testing where possible, and examine the results separately for:

  • New customers
  • Existing customers
  • Frequent purchasers

Customer status can materially change the answer.

Prospecting

Prospecting can consume a large share of media investment and may affect customers earlier in the purchase journey.

Geo experiments can be useful when user-level randomization is difficult or when several channels operate together.

Test results can also provide calibration points for broader media mix models.

Lifecycle Email And SMS

Owned channels are where CDP-defined holdouts are strongest because the enterprise controls delivery.

Randomize at the customer level, suppress the control across the relevant owned channels, preserve assignment, and measure incremental outcomes.

The fundamental method is the same, but the control is cleaner than an exclusion-list holdout sent to an external ad network.

Where CDP Data Actually Helps Incrementality Testing

A CDP should support the experiment rather than being presented as the experiment itself.

Define Eligible Audiences By Customer Status

Incrementality often differs between new, repeat, and loyal customers.

The eBay study itself found meaningful heterogeneity: new and infrequent users responded differently from frequent users.

The CDP is often the system with enough cross-channel history to classify those customers consistently.

Supply Outcomes The Ad Platform Cannot See

A media platform may not see:

  • Store purchases
  • Call-center sales
  • CRM-validated opportunities
  • Subscription renewals
  • Loyalty activity

CDP and warehouse data can join those outcomes to experimental groups.

Maintain Owned-Channel Assignment And Exposure

For channels the enterprise controls, the CDP can record:

  • Treatment assignment
  • Holdout assignment
  • Exposure
  • Experiment membership
  • Outcome history

That prevents one campaign from accidentally invalidating another experiment.

Feed Results Back Into Marketing Decisions

Experiment results should update:

The value comes from changing decisions, not accumulating lift-study reports.

Know Where The CDP Stops

A CDP does not:

  • Randomize users inside Google or Meta
  • Guarantee an ad platform honors an uploaded exclusion
  • Perform the statistical analysis automatically
  • Eliminate the need for experimental design

That boundary is important because overstating what the CDP controls leads directly to invalid paid media experiments.

Use Incrementality With Attribution And Media Mix Modeling

The three methods solve different problems.

Attribution Supports Daily Optimization

Attribution remains useful for understanding the journeys and touchpoints associated with conversions.

It is fast and granular.

Its limitation is causal interpretation.

Incrementality Answers Specific Causal Questions

Incrementality asks whether a particular treatment changed behavior.

It is stronger for decisions such as:

  • Should we keep this retargeting spend?
  • Does brand search create additional conversions?
  • What is the incremental return from this platform?

Media Mix Modeling Supports Portfolio Allocation

Mix models evaluate channel contribution over time using aggregate variation.

They can help answer broader budget questions when user-level experimentation across every channel is impractical.

Incrementality results can then be used as calibration evidence for the model.

The strongest measurement program therefore does not search for one universal truth system.

It triangulates.

A Seven-Step Incrementality Testing Method

Step 1: Define The Decision And Hypothesis

State exactly what will change if lift is high, low, or inconclusive.

If the result cannot change a decision, reconsider whether the test is worth running.

Step 2: Choose A Design That Can Create A Real Control

Prefer platform-run randomization or geo testing for paid media.

Use CDP-controlled holdouts where the enterprise controls delivery.

Step 3: Size And Time The Experiment

Define:

  • Baseline
  • Minimum detectable lift
  • Sample requirement
  • Conversion lag
  • Planned test duration

Do this before launch.

Step 4: Define Eligibility And Assignment

Use CDP customer status where useful.

Where the enterprise controls randomization, assign using a stable identity so one person cannot move between treatment and control across devices.

Step 5: Run And Monitor Without Repeatedly Peeking

Monitor operational failures, leakage, and collisions with other tests.

Do not repeatedly stop and restart the experiment based on early movement in the results.

Step 6: Analyze The Result With Uncertainty

Report the lift with confidence or credible intervals appropriate to the method.

Break results down by preplanned customer segments.

If the experiment lacked sufficient power, describe a flat result as inconclusive rather than declaring the channel ineffective.

Step 7: Act And Record

Translate the result into:

  • Budget changes
  • Suppression changes
  • Audience strategy
  • Future experiment priorities
  • Mix-model calibration

Record the design and outcome so the organization does not repeat the same question without using what it already learned.

How Stable Kernel Approaches Incrementality Testing

Stable Kernel approaches incrementality as a measurement architecture problem that connects experiment design, customer data, paid media, and business outcomes.

Choose The Design Before Building The Segment

Stable Kernel can help enterprise teams determine whether a question is better answered by platform Conversion Lift, geo experimentation, an owned-channel holdout, or another defensible comparison.

The CDP can then supply the eligibility, customer-status, exposure, and outcome data the chosen design requires.

Measure Leakage Instead Of Assuming The Holdout Worked

For paid media experiments using customer-list exclusions, Stable Kernel can help teams quantify how much of the intended holdout actually reaches the advertising platform, how long the exclusion takes to become active, and whether identity or consent gaps contaminate the test.

If your growth team is using attributed conversions to make major media decisions, or using CDP-defined paid media holdouts without knowing how much of the control group is actually excluded, Stable Kernel can help design the experiment, size the test, measure the leakage, and connect the result to budget and suppression decisions.

Request an incrementality design and leakage review from Stable Kernel to determine which paid media questions need platform lift, geo testing, or customer-level holdouts and whether your current customer data can support a valid experiment.

Three Questions Before Your Next Incrementality Test

1. Can This Design Actually Withhold The Treatment?

If the test depends on an uploaded paid media exclusion, measure match coverage and processing latency before calling the group a control.

2. Is The Test Large Enough For A Flat Result To Mean Anything?

If the experiment cannot detect the amount of lift that would change the business decision, increase the sample or redesign the test.

3. What Will We Do Differently After The Result?

Write the budget, audience, or suppression decision before launch.

A “no” to any of these questions identifies what needs to be fixed before the experiment starts.

FAQ

What Is Incrementality Testing In Marketing?

Incrementality testing estimates the causal effect of marketing by comparing outcomes for a group that could receive the treatment with a comparable group that could not. The difference is incremental lift. It answers whether the conversions would have happened without the marketing rather than simply assigning credit after they occurred.

How Is Incrementality Testing Different From Attribution?

Attribution assigns credit for conversions associated with marketing touchpoints. Incrementality testing creates a counterfactual through a control group and estimates which conversions the marketing actually caused. Large field experiments have shown that attributed and causal results can differ substantially, but the size and direction of that difference depend on the advertiser and situation.

How Do You Run An Incrementality Test For Paid Media?

Use a design that can create a credible control, such as platform-run Conversion Lift, a ghost-ad approach, or a geo experiment. Size the experiment based on the baseline and minimum effect you need to detect, run it long enough to capture conversion lag, and interpret the result with uncertainty intervals. CDP exclusion-list holdouts are usually less reliable for paid media because some intended control users can still see the ads.

Can A CDP Hold Out Audiences From Ad Platforms?

Only partially. A CDP can upload an exclusion audience, but the advertising platform can exclude only the customers it successfully processes and matches, and consent restrictions can remove additional customers from the uploadable population. Google currently says Customer Match processing can require up to 24 hours, so paid media exclusion-list holdouts should be monitored for leakage.

How Big Does An Incrementality Test Need To Be?

It depends on the baseline conversion rate and the smallest effect worth detecting. As an illustration, detecting a 10% relative lift from a 5% baseline with approximately 95% confidence and 80% power requires about 31,000 users per arm under simple user-level randomization. Smaller effects require much larger samples, so a flat result from an underpowered experiment does not prove the advertising had no effect.

Is Brand Search Or Retargeting Incremental?

It depends on the advertiser. eBay experiments found little incremental short-term value from branded search in its environment, while later research across many brands found competitors can capture traffic when the focal brand stops bidding. Retargeting likewise reaches customers who may already be on a purchase path, so both tactics should be tested rather than classified as automatically incremental or wasteful.

How Does Incrementality Testing Relate To Media Mix Modeling?

Incrementality tests answer specific causal questions about individual campaigns, channels, or spend changes. Media mix models estimate contribution across a broader portfolio using aggregate variation. Experiment results can provide calibration evidence for the mix model, while attribution can continue supporting faster day-to-day optimization.

Can Stable Kernel Help Design Incrementality Tests With CDP Data?

Yes. Stable Kernel helps enterprise teams select the right incrementality design, define eligible audiences and outcome data, estimate test requirements, evaluate CDP identity quality, measure paid media holdout leakage, and connect results to budget and suppression decisions. The objective is to make the experiment useful enough to change a real growth decision rather than simply generate another measurement report.