Quick answer: A holdout group is a randomly selected slice of your audience that is deliberately kept from seeing a campaign, so you can compare its results with the exposed group and measure how many conversions the ads actually caused.
A holdout group is the “no ads” side of a controlled experiment. You split an audience at random. One part sees your campaign as normal (the test or exposed group). The other part is held out and sees nothing from that campaign (the holdout group).
The gap between the two groups is called incremental lift. It tells you how many conversions happened because of your ads, and how many would have happened anyway. Platform reports cannot tell you that on their own. A holdout group can.
Why holdout groups matter
Ad platforms report conversions that follow an ad view or click. They do not report whether the ad caused the purchase. Someone who already planned to buy from you, searched your brand name and clicked a retargeting ad still shows up as a conversion.
A holdout group is the only way to separate sales your ads caused from sales that were going to happen regardless. This matters most for retargeting, brand search and any campaign that mostly reaches people who already know you. These campaigns often show the best ROAS in the dashboard and the weakest real impact.
The industry is moving the same way. A 2025 research paper on Amazon Ads measurement notes that the IAB and the Media Rating Council recommend randomized controlled trials for measuring advertising incrementality in their 2024 retail media guidelines. A holdout group is the control arm of that kind of trial.
If you never run one, you are budgeting on credit that the platform assigned to itself. That is how brands keep scaling campaigns that add very little revenue.
Holdout group vs A/B test control
Both use a control group, but they answer different questions.
| Attribute | Holdout group test | A/B test |
|---|---|---|
| Control group sees | No ads from the campaign | A different version of the ad or setup |
| Question answered | Do these ads add any sales at all? | Which version performs better? |
| Main output | Incremental lift, incremental conversions, iROAS | Winner between variants |
| Cost | Missed impressions on the held-out audience | Usually none, both groups get ads |
| Best used for | Budget decisions, channel validation | Creative, audience and bid optimisation |
How to set up a holdout group and calculate lift
The steps are the same on every platform, and only the tool changes.
- Pick one question. For example: “Do our Meta retargeting ads add sales?” One test, one campaign type.
- Randomise the split. Assign users (or regions, for a geo holdout) to test and holdout at random. Platform lift tools do this for you.
- Choose the holdout size. Google Ads allows a holdback between 1% and 50% for user-based Conversion Lift. A bigger holdout collects data faster but costs more missed impressions. A smaller one needs a longer test.
- Set the duration. Cover your average conversion lag, meaning the time between seeing an ad and converting. Google allows studies as short as 7 days and recommends running them for at least 14.
- Do not change the campaign mid-test. Creative swaps and audience edits muddy the result.
- Compare the groups. Use the formulas below.
Formulas
Incremental conversion rate = test conversion rate – holdout conversion rate
Lift % = (test rate – holdout rate) / holdout rate x 100
Incremental conversions = test conversions – (test users x holdout rate)
iROAS = incremental revenue / ad spend
Worked example (₹)
An online store runs a retargeting campaign. It spends ₹9,00,000 and holds out 10% of the audience. Average order value is ₹4,000.
| Group | Users | Conversions | Conversion rate |
|---|---|---|---|
| Test (sees ads) | 90,000 | 2,700 | 3.0% |
| Holdout (no ads) | 10,000 | 200 | 2.0% |
- Lift: (3.0% – 2.0%) / 2.0% = 50%.
- Conversions expected without ads: 90,000 x 2.0% = 1,800.
- Incremental conversions: 2,700 – 1,800 = 900.
- Platform view: 2,700 conversions x ₹4,000 = ₹1,08,00,000 in revenue, or 12x ROAS on ₹9,00,000.
- True view: 900 x ₹4,000 = ₹36,00,000 in incremental revenue, or 4x iROAS.
In this example, one third of the conversions the platform would claim were caused by the ads, and the true return is 4x, not 12x. The campaign still pays off, but the budget case looks very different. (These numbers are illustrative, not client results.)
Common mistakes with holdout groups
- Splitting the audience by hand. Picking “engaged users” or “one city” for the holdout builds bias into the result. Use random assignment or a platform lift tool.
- Running the test too short. If buyers take two weeks to convert and the test runs for one, you miss conversions and understate lift. Google reports up to a 17% drop in absolute lift for long-lag studies that run under 14 days.
- Testing too little volume. Low conversion counts produce noise. Google’s user-based Conversion Lift requires at least 1,000 observed conversions and a minimum campaign budget of $5,000 USD.
- Measuring one campaign while others still reach the holdout. If the held-out users see your other ads, the test understates the value. Google recommends including all campaigns of the same type, audience and targeting strategy.
- Editing the campaign during the test. Changing creative or audiences mid-study makes it hard to know what caused any difference.
Frequently asked questions
What is a holdout group in marketing?
A holdout group is a randomly chosen part of your audience that does not see a campaign during a test. Its conversion rate shows what happens without your ads, so the difference against the exposed group is the lift your ads caused.
How big should a holdout group be?
Big enough to detect the lift you expect. Google Ads allows a holdback of 1% to 50% and recommends a size when you set up the study. Larger holdouts collect data faster but cost more missed impressions, while smaller holdouts need longer tests.
How long should a holdout test run?
Long enough to capture your average conversion lag. Google Ads allows studies as short as 7 days and typically recommends more than 14 days, with 14 as the minimum for reliable results.
Is a holdout group the same as a control group?
In ad measurement, a holdout group is a control group that receives no exposure to the campaign. In an A/B test, the control group usually still sees ads, just the current version, so it is not a holdout.
Can small advertisers run holdout tests?
Native tools have limits. Google’s user-based Conversion Lift needs at least 1,000 observed conversions and a $5,000 USD minimum budget, and it is not available to every account. Smaller accounts can run a geo holdout, where some regions get no ads, but the results are noisier and should be treated as directional.
How Uginix helps
Most agencies report the platform’s ROAS and stop there. Uginix sets up holdout and geo-holdout tests inside our analytics and attribution work, picks a holdout size your volume can support, and reports incremental ROAS next to platform ROAS in plain language. If a campaign turns out to add little, we tell you and move the budget.
Want a second opinion on your ad account? Uginix does a free audit, no lock-in, no pitch deck. Book it here.
Related terms
- Incrementality testing: measuring the sales your marketing caused, not just the sales it was present for.
- ROAS: revenue earned for every ₹1 spent on ads.
- Attribution window: the period after an ad interaction in which a conversion is credited to that ad.
- Geo holdout test: a holdout where whole regions, not individual users, are withheld from ads.
Sources: Google Ads Help, About Conversion Lift, Google Ads Help, Conversion Lift based on geography, Gordon, Moakler and Zettelmeyer, Marketing Science (2023), Blake, Nosko and Tadelis, Econometrica (2015).