Friday, August 12th, 2011

Ads Work!

Posted by: Tom Phillips

One of our brilliant new data scientists decided to see whether some of the work he had done in epidemiology to determine the efficacy of disease treatments could be adapted to measure the impact of advertising.  What started as an academic exercise has produced a treasure trove of insights into the impact of digital advertising, and how ad effectiveness varies with targeting methods and creative strategies.
But as excited as we are about the potential of the new methodology to provide insights into our own work identifying high-performing audiences for major brands, the real headline is more fundamental and earthshaking.  ADS WORK!

In 29 of the 30 campaigns we measured, we saw a material lift in the site visit rate of browsers that viewed the ad of our client (more about that one outlier later).  Again and again, we hear – to our dismay – that because click rates on digital ads are microscopic, the ads don’t work.  Well, there is bountiful research showing that clicks alone are an incomplete and even misleading measure of impact.  But beyond that non-negative finding, which our own research has substantiated, we have very positive evidence that digital ads work, and that they deliver lift varying from material to magnificent.

I’m going to defer to our brilliant data scientist – Ori Stitelman – to describe his methodology.  Suffice it to say that he has developed an alternative to A/B testing; the issue with standard A/B testing is that it tends to be both noisy and expensive.  The new methodology is called Non-Invasive Causal Estimation (“NICE”).  NICE is a non-invasive analysis that uses mathematical models to calculate the likelihood of populations to convert, comparing those exposed to an ad to those not exposed.  The models constructed control for all confounders (factors other than ad exposure that distinguish the two populations), including the fact that our targeting technology tends to serve ads to people who are more likely to convert.

The chart below details the lift we measured from running campaigns for thirty clients across two segments – prospecting and retargeting.  These results capture the impact of our ads, not all ads from that marketer.  As such, the effect would likely be higher if the client limited the number of vendors used for a particular campaign.

The results are astounding.  For each of thirty clients, we looked at site visit lift for two populations – prospecting candidates and retargeting candidates.  The average lift for the prospecting candidates was 90% (median 70%).  The average lift for retargeting candidates was 20% (median 10%).

Translated, that means that serving ads to the people that we had identified as good prospects for the brand (but not current customers) caused them to nearly double their site visit rate!  No tricks, and no need for clicks.  The ads had impact, and eliminating all other factors, we found that digital ad exposure resonated with digital media consumers.

As indicated above, we had one case where the lift was zero for both prospecting and retargeting populations (see right side of chart above).  When we investigated this particular travel client, we found that the issue appeared to be in the creative.  Driven perhaps by the desire to maximize clicks, the creative team had generated an ad that highlighted sweepstakes over brand.  In fact, the brand itself was buried – just a thin line of text in the bottom right with a tiny logo.  No wonder we observed no lift in site visits from the population viewing this ad!

Perhaps the most dramatic result was achieved with three of our telecom clients – all major marketers with huge budgets.  For these three, the relative lift for the prospecting segment averaged 106%, while the relative lift for the retargeting segments averaged 3%.  And the additive lift (the gross increase in site visit rates) for prospecting segments exceeded the additive lift for retargeting segments in two out of three cases.

We are just getting started using the NICE methodology to prove the efficacy of digital advertising and improve the results we can deliver.  For now, we can happily report that in case after case, digital display advertising is having impact, and that that impact is particularly robust when our M6D prospecting technology is used to deliver the audiences most receptive to a marketer’s brand message.