To find out what an ad channel adds to sales, you switch that channel off or turn it down in some regions, or for a period, and compare with the rest. It is the simplest incrementality test when customers cannot be split one by one, and it works on a mid-sized company’s budget. The platforms’ own studies, such as Google’s Conversion Lift, serve large accounts. For everyone else, a well-designed geo test or on/off test is enough to decide where the money goes.
Why attribution is not enough
Ad platforms count as theirs the sales of anyone who clicked an ad before buying. In campaigns that talk to people who already know the brand (brand search, retargeting, customer lists), a good share of those people would have bought without the ad. Attribution cannot tell the two cases apart, and a test with a comparison group can.
The full explanation is in incrementality: how to know what your marketing actually causes.
The geo test
How it works
The regions where the brand sells are split into two groups with similar sales. In one group, the channel stays as it is, and in the other it is switched off or spend is reduced. After a few weeks, the sales trend in both groups is compared, against the trend each one had before the test.
In Portugal, the split is done by district or by region. Lisbon and Porto carry a lot of weight, so they should not sit in the same group. Platforms allow campaigns to be targeted by location, and online shop orders carry a delivery address.
When it fits
When the channel cannot be split by person (paid search, YouTube, Meta with broad audiences) and the brand sells in enough regions to form two similar groups. It works best when sales have steady weekly volume.
Points to watch
- Regions are not identical. You compare each group’s change against its previous period. If the test group sold 10% less before and keeps selling 10% less, the channel was not adding anything.
- There are leaks. Someone from Setúbal sees the ad in Lisbon. The measured effect comes out slightly below the real one, and the result should say so.
- Everything else stays the same. If a promotion launches in one of the groups during the test, the test loses its value.
The on/off test
How it works
When there are not enough regions, you split time instead. The channel stays on for two weeks, off for two, and so on, for six to eight weeks. The average of the weeks on is compared with the average of the weeks off.
When it fits
In small accounts, in a single market, and for specific questions. The most common case is brand search. If sales barely drop in the weeks the campaign is off, people were arriving through the organic result just below the ad.
Points to watch
Time has effects of its own, such as the start of the month, public holidays or the weather. That is why several short periods are alternated instead of one long one, and weeks with known events are avoided.
Platform studies
Google and Meta offer incrementality studies run by the platform itself, which split users at random between seeing and not seeing the ads. Google’s Conversion Lift is the best known. They are reliable by design, but they need conversion volume that only large accounts have, and they only answer for the platform running them.
How to read the result
Before starting, write down what will be measured, for how long and what each possible result will lead to. For example: “if sales in the regions without paid search fall by less than 5%, we cut brand search spend in half”. With the rule written beforehand, the result decides on its own. If it is written afterwards, everyone reads the result their own way.
The result is read on margin. A channel that brings incremental sales at a negative margin is still a bad investment. How to calculate margin per order is in what is contribution margin, and the media plan these tests fit into is in paid media strategy for scaling brands.
What to do with several tests
One test answers one question. After a few, the company knows what each main channel adds and can move budget on that basis. With enough history and budget, the results can calibrate a marketing mix model, which estimates the effect of every channel at once. Without the tests, the model is only an estimate.
Tests are not run during the weeks of large campaigns, which are measured differently, as we explain in how to measure Black Friday.
Frequently asked questions
What is a geo incrementality test?
A test in which an ad channel is switched off or reduced in some regions and the sales trend is compared with the regions where the channel stayed the same.
How long should it last?
Four to six weeks for a geo test, and six to eight for an on/off test, alternating two-week periods.
Does it work in a small country with few regions?
Yes, if sales are enough to form two similar groups and the largest cities sit in different groups. When there is not enough volume, the on/off test is the alternative.
Should I switch off brand search?
It depends on the test result. A good share of the sales credited to that campaign may arrive anyway through the organic result, and only the test tells you how much.
Want to know what your paid search actually adds?
Book a call and we will design the test with you, with the decision rule written down beforehand.
