The gap between being able to measure incrementality and being able to use it is larger than you think. A new IAB EU and US report deconstructs incrementality measurement and helps advertisers and Retail Media Networks get aligned on what really matters.
There are two topics that are discussed at every retail media conference: measurement and incrementality. They are two different things – but are obviously intertwined.
Incrementality measurement is trying to understand which advertising actually drives sales. Another way to phrase this is sales that would not have occurred without a specific marketing activity.
A new report jointly issued by the IAB EU and US defines incrementality as “measuring the causal impact of marketing by identifying the additional business outcomes directly driven by a campaign or tactic, compared to what would have occurred in the absence of marketing activity.”
Why this matters
Understanding incrementality is important for both retailers and advertisers for a number of reasons:
- Advertisers prefer not to pay for outcomes that would have occurred anyway – hence measurement of incremental results is seen as important.
- Delivering results to advertisers are important for retailers to get repeat media sales.
- Access brand marketing budgets.
The lure of brand budgets – arguably the largest marketing budget ‘pot’ in CPG brands – is dependent on having measurement that shows results. These budgets typically flow through media agencies (and large advertisers are dependent on these agencies for execution) and they spend a large amount of money working out if this spend is optimised and delivering a return on investment.
These budget ‘pots’ won’t automatically flow to Retail Media Networks as too much of the focus in retail media is on activation and ad-operations, not outcomes. For many brands and retailers, success is often measured in getting something live, then reporting a few surface-level results, not outcomes.
That was fine when retail media was in its fast growth phase. All industries mature, and for retail media to move from ‘growth phase’ to ‘BAU’ = business as usual, embedded in marketer’s media schedules as a matter of course, proof of results becomes even more important.
Measuring incrementality
Experts all have their own opinions on which methods are best when measuring incrementality, but they all involve a combination of testing and experimentation.
- Geo holdouts: Run campaigns in some regions while keeping others as control, then compare sales lift.
- A/B testing: Split audiences randomly into test (exposed) and control (unexposed) groups, then measure the difference in outcomes.
- Pre/post analysis: Compare sales before and after a campaign, adjusting for seasonality and trends.
- Matched market testing: Select similar markets, run media in one but not the other, and analyse the lift.
- Media Mix Modelling (MMM): Statistical modelling to quantify the impact of media spend while accounting for other marketing factors.
For digital media, A/B testing is the most rigorous for measuring true incrementality, directly measuring causality, but is harder to execute at scale.
The challenges of measuring incrementality
There are two sets of challenges around incrementality. The first set of challenges are structural or external barriers; the second set is around interpretation.
Here’s the external or technical challenges that barriers that stop you from being able to measure incrementality properly in the first place:
- Lack of data or resources to measure incrementality.
- Lack of agreement of how incrementality should be measured.
- Lack of communication on how incremental sales are lower than ‘attributed’ sales.
- Lack of the right tools: econometrics is expensive and all depends on your choice of baseline demand.
- Measuring if advertisers are getting new or repeat buyers varies by cookie window!
In other words, incrementality measurement ain’t easy! Even if we can solve the operational challenges, the next barrier is about usefulness – the internal or practical questions you ask once you’re trying to apply incrementality measurement.
For example:
- At what granularity is retail media being incorporated into the model? Channel, platform, ad-type, targeting type, campaign?
- What level of incremental analysis makes sense (keyword/SKU/category)?
- What sales channels are included in the model? Is there a separate model for ecommerce or a specific retailer’s sales?
- Are we controlling for ecommerce / marketplace factors which may influence media performance? Organic vs. paid search rank, SKU level promotion, SKU level pricing?
- Should measurement be at shopper level?
- What would we actually do with the answer if we know if?
- Are we as an advertiser set up to act on it?
- When is “close enough” good enough?
- Should we even compare retailers on the same basis?
In other words, even if we can measure incrementality, what does it actually mean — and what do we do with it?
The IAB Guidelines for Incremental Measurement in Commerce Media points some of this out: "the challenge marketers face today with incrementality is two-fold: choosing the right method and understanding the claims it supports and its causal reliability."
The paper points out that the "usage of incrementality as a measurement approach should be guided by the business goal in mind – not by the availability of a specific measurement method. Different questions require different levels of causal rigour, data access, and time horizons."
This last point is one that should be quoted by participants on panels when talking about retail media: incrementality measurement is NOT obligatory, and not all methods are equal – and sometimes a simple measure will do.
What is NOT incrementality?
At the aforementioned retail media conferences, there are lots of words used talking about measurement – and these are often mixed up with incrementality measurement. Here are just some examples of traps we all fall into when talking about measurement and incrementality as pointed out by the IAB:
- Attribution: Attribution assigns credit, not causality.
- Media-attributed ROAS: Return on ad spend is a directional efficiency metric, not a measure of impact.
- Correlation: Sales increases during a campaign does not mean the campaign caused the increase. Sales may rise due to seasonality, promotions, price changes, and/or market or brand dynamics.
- Pre/post testing: Comparing performance before and after a campaign without accounting for external factors can overstate or understate true impact.
- Historical performance analysis: Prior performance is helpful for context but it cannot isolate what would have happened without the campaign.
Should brand advertisers care about retail media?
Back to the point in the IAB paper: "usage of incrementality as a measurement approach should be guided by the business goal in mind – not by the availability of a specific measurement method."
In my (subjective) experience as a marketing director for many years, getting into the minute details of measurement simply was not worth the effort, except for the large, broad campaigns where a lot of money had been invested in TV creative and media.
Even the large-scale econometric measurement models that I used were always subject to a caveat: what was the base level of sales that would have happened anyway? This is at the heart of the challenge with any media mix modelling or econometrics.
The second challenge is the sheer scale required to make the models work. For example, a media mix modelling programme may require collection of two to three years of data at a weekly level. This data would undergo cleaning, standardisation, and transformation in preparation for modelling. Variables or features for the model would be tested, selected, and undergo further tuning and relevant coefficients evaluated and updated annually or bi-annually.
This is the key point for marketers – don’t get carried away with incrementality being seen as a "must have". Talk of "gold standard" measurement can also lead to "gold standard costs!"
Surprising as it may seem, incrementality is not the ultimate goal for many brands. Given it can be both hard and expensive to measure, "the juice may not be worth the squeeze."
I have heard brand advertisers say version of the following:
- "We want to deliver unmissable content to the right shoppers, at the right time" – they see retail media as an opportunity to get messages in front of shoppers
- "We don’t need to spend time on measurement, they show up in our sales numbers": in other words, if the campaign is good enough, the results should be VERY clear in the normal weekly and monthly reporting.
- "We want to be a partner to our retail channel" – the results are secondary to the relationship with the retailer.
The message to advertisers on Retail Media Networks is this: if you have the data, the resources and the capabilities, you should pursue incrementality measures. If you do not, work with easy-to-use measures such as A/B testing, experiment-based approaches and proxy tests.
Speed is often the key requirement for marketers – and directionality reads for "inflight tuning" – as the IAB call it – can be good enough. Or, you could just look at the sales results: as a marketing director, you should be able to look at weekly sales results and get a sense if a campaign is working as expected for many campaigns.
One quote from a major dairy brand when talking about a retail media campaign recently was this: "I don’t have to look at detailed measurement results, I can just look at my sales reports and know immediately."
The IAB report also point this out clearly: "all methods are equal in their ability to establish causality, and not all use cases demand the same level of rigor. Marketers should align the method to the decision at hand:
- For high-stakes strategic questions – such as budget allocation, cross-channel planning, or proving ROI to the C-suite – use methods with stronger causal foundations.
- For fast, tactical optimisation or operational decisioning, lighter-weight proxies may be acceptable, provided their limits are acknowledged."
Should Retail Media Networks care about incrementality?
All of the use cases and measurement types look complex. Regardless of what the brand advertisers might think, Retail Media Networks should take a different view.
Understanding incrementality is important for the growth of retail media for a number of reasons.
1. Retail Media is intrinsically built differently so Retail Media Networks need to be able to show this: Retail Media promises ‘closed look measurement’ as part of its USP, so proving this is paramount.
Pete Robins of Project 5 points out that "if retailers can track sales with precision, opinions matter less as brands can run structured experiments, and planners can use data engineering and AI to model scenarios, test hypotheses and codify rules of investment. This creates repeatable blueprints that drive business outcomes, not just media outputs." In other words, it is in the interests of RMNs to show things are working – as they can get repeat business.
2. Competition from other channels: Marketers have lots of choices for their media budget. TV, radio, outdoor, Google, Meta, TikTok – the choices are endless and increasing. The question that Retail Media Networks should be answering is not "what can I buy?" but "why should I buy it?"
As Josh Clarkson, Global Retail Media lead at Mars and CEO of Edify points out, "Marketing budgets aren’t infinite… and advertisers legitimately ask 'why should I buy it instead of buying other (non-retail) media which can deliver the same outcome/better outcome/cheaper/more measurable/already in my MMM?'"
3. RMN selling channels not solutions to marketing problems: Retail Media Networks often focus on selling ‘channels’ rather than selling true creative marketing campaigns. Sean Crawford, Managing Director of SMG North America points out that "Retail Media campaigns should be intrinsically tied to wider brands marketing, rooted in data with upfront media planning that enables full funnel campaigns onsite, offsite and in-store." Stop selling ad units and sell solutions to problems.
But these reasons are not the only reasons. Perhaps there is a bigger prize. Perhaps retail media can deliver on a bigger promise than just better execution of existing marketing plans such as driving sales, basket size, upselling or cross-selling.
Measurement can help deliver on the real promise of retail media
Breakthrough outcomes including creating and capturing more value through retail media, shopper, brand insights can radically change the category for the brand and the retailers.
Christy Ramroop, senior director, commerce at Flywheel points out that "the real opportunity lies in connecting Retail Media to the full business ecosystem — modelling how media investment influences supply chain, product allocation, pricing power, inventory velocity, innovation and assortment."
With this thinking, Retail Media becomes not a ‘side of the desk’ activity, but instead one that is intrinsically tied in with the fortunes of the retailer and their supplier brands. Now that is a form of incrementality I can get behind!
Originally published on InternetRetailing