Measure What Matters Part II: Incrementality

As an advertising agency with chops in paid media, public relations, search marketing, affiliate marketing, social media, influencer marketing, creative and content, and data analytics, we get measurement. In this content series, we’re delving into the nuanced and multifaceted measurement process to help marketers understand which measurement solutions are best for their brand and how they can be integrated into a holistic measurement strategy. Read more about why it’s important to Measure What Matters in our introduction piece.

Last time, we covered Media Mix Modeling (MMM). Here, we’re digging deep into incrementality, which removes the bias and shortcomings of platform-based attribution models (that often tilt the scales in their favor) and identifies where your marketing dollars are actually having an impact and where they are not. 

What is incrementality? 

Put simply, incrementality is the sales and revenue that would not occur without the influence of a particular media tactic. Often, some of the conversion volume that is attributed to specific media would still occur without media in market, and customers that happened to be exposed to advertising would still have purchased even if they didn’t see the ad.

Picture this: you saw a product on an Instagram ad and then heard your friend talking about it. A few days later, you find yourself at a store where there is an advertisement for the same product. You purchase the product and the last advertisement you saw gets credit for converting you to a customer, when in reality, there were several elements that impacted your decision. 

Image designed by Backbone, inspired by the Marketoonist.

In these cases, media platforms are taking credit for capturing pre-existing demand (e.g., users that were going to buy anyway). Incrementality refers to transactions and revenue that were casually impacted by media. 

Incrementality is measured through experimentation. Most commonly, a specific type of media is withheld from one or more test audiences, while a control audience continues to receive media exposure. Typically, test and control audiences are grouped by separate geographic regions but can also be defined by unique, known audience groups on certain platforms. The difference in online transactions from all sources between test and control cells defines the causal impact of the tested tactic to sales. This gives us an estimated portion of total transactions in our control cell that are incrementally driven by the tactic. Comparing that result to the number of transactions claimed by the media platform defines the incrementality and paints a more accurate picture of conversions across media.

What questions does incrementality answer? 

Incrementality primarily answers the question: how much of the conversion volume the tactic takes credit for is considered incremental?

In many cases, this methodology can quantify the extent to which low-funnel, active demand channels might overstate conversions or return on investment. However, incrementality can also assess the impact of removing or adding a particular media tactic, quantifying how much conversion volume would be lost or gained as a result. 

Incrementality is most informative when applied to cost efficiency metrics. Incremental return on ad spend (iROAS) and cost per acquisition (iCPA) weigh results claimed by the platform, offering a new perspective on cost efficiency.

Incrementality is measured at a point in time, so variables like spend, seasonality, and changes to platform attribution models will affect results. Testing often and in varying conditions can pinpoint a more holistic idea of incrementality. For instance, incrementality will likely change as spend is scaled. Testing at both a lower and higher spend level can assess the impact of increasing investments in a particular media tactic.

What does incrementality measure and output? 

• Incrementality Factor: an estimated percentage of platform-reported conversions that are incrementally caused by a media tactic
• Incremental Return on Ad Spend (iROAS): The amount of incremental revenue per dollar of media
• Incremental Cost per Acquisition (iCPA): The media cost per incremental transaction

What does incrementality not measure or output? 

• Direct attribution: incrementality results are experimental and do not track direct exposure to conversion results, by design
• Media mix scenarios: individual tests only measure incrementality at the specific media mix that existed during the test, modeling results from multiple tests can indicate some impact from varying media mixes but is less informative than a media mix model (MMM)

What is necessary to implement incrementality? 

• Organization
  • Rigorous testing and modeling procedures
  • An appetite to test and test often
  • Willingness to absorb small impacts to overall sales or ROI
  • Decision making strategies developed before test completion
• Personnel
  • A trusted partner or in-house statistical and media testing expertise
  • Marketing strategists well versed in interpreting and acting on results
  • Media buyers capable of executing geo-based targeting and exclusion strategies
• Data and Technical Resources
  • Connections to geo-level platform and web analytics data
  • A results archive and or model to track incrementality across tactics and individual tests

How can incrementality configure and validate other measurement tools? 

• Media mix model (MMM)
  • Incrementality factors can be applied to MMM results if platform conversion metrics are factored into the model.
  • Incrementality results can identify recent changes to tactic effectiveness that may take longer to realize with an MMM.
• Multi-touch Attribution (MTA)
  • MTA model weights can be validated against incrementality results. Channels receiving significant credit should have high incremental transactions, and vice versa.
  • In places where incrementality is low but MTA credit is high, the MTA model may be overstating credit for tactics that are capturing existing demand.
• Post-purchase survey (PPS)
  • Customer-reported media touchpoints in PPS can be used to validate incrementality. Channels with high incremental transactions should be mentioned frequently in survey questions concerning where customers heard about the brand or product.

Specifically, it’s an estimated percentage of platform-attributed transactions estimated to be caused by a media tactic.

Backbone incrementality case study

Challenge

One of Backbone’s apparel clients was running a paid social, mid-funnel campaign with a high return on ad spend (ROAS) of $10.04 as measured by the media platform. 

Was the return on this campaign actually this lucrative? Or was the platform overstating its true, causal effect on conversions by driving already intending purchasers to the site?

Solution

Backbone executed a geo holdout test, pausing the paid social campaign in five states for several weeks. These states were selected due to the similarity of their historical performance to states comprising the control cell where the campaign continued to run.

Results

Test results indicated the only 17% of the conversion volume that the platform attributed to the campaign was incremental. In other words, an estimate 83% of transactions and revenue would still occur via other channels without the presence of the campaign.

This incrementality factor changed perceptions of the campaign’s cost efficiency. When applied to the cost metrics of the campaign from the time period preceding the test, iROAS was $1.70 and iCPA was $90.11. These two values differed significantly from the results measured by the platform. While the tactic still provided value by generating consideration and producing some conversions, it didn’t warrant a massive budget increase due to its modest incremental cost efficiency.

Overall, incrementality is crucial for understanding how platforms and media types are actually converting customers. Experimenting with incrementality can reframe expectations on the ROI of media tactics and inform which are worth increasing or decreasing investment in. If you have any questions about incrementality or want to explore it for your brand, please reach out to us at info@backbone.media. 

Stay tuned for part III of this series: Post Purchase Survey (PPS).