20th Century retailer John Wanamaker once famously said, “I know half my advertising is working. I just don’t know which half.”
You can’t lay down that kind of challenge at a Marketing Analytics Conference and not get some real answers to this ageless question! An analytics and testing success story was shared by Michelle Ogle, Associate Director, North America Digital Marketing for Starwood Hotels and Arun Kumar, Group Director of Analytics for Razorfish at the conference earlier this month, where I was privileged to serve as the Program Chair.
“Incrementality” helps Starwood understand the incremental value of digital display performance media programs to acquisition results. To uncover the value, Michelle worked with Razorfish to run a PSA Test – often considered the gold standard for display testing. What the PSA test does is replace Starwood ads with public service announcements to a control group to measure the lift from the ads.
The steps taken to prepare for the test were:
- 5% of all Sheraton and Westin cookies were designated as controls.
- These were cleansed of all ads for one week to account for laggard impact. “We paused all other Sheraton and Westin campaigns, including upper funnel branding campaigns so they would not color the test,” Arun explains.
- During the test, the control group was shown a PSA ad instead of Starwood ads.
“The question the test is designed to answer is, ‘If I don’t show my audience any ads, what are the chances I will get a booking?’” Arun said. “The difference between test and control is defined as the ‘incrementality’ of this project.”
Running PSA ads on an actual media buy is not an insignificant investment. “We had to utilize 5% of the Starwood cookies in learning this information. It was an interesting conversation with our brand owners,” Michelle said. “We asked our brand stakeholders to take a portion of active media budgets meant to sell hotel rooms and instead put it behind PSA ads. They all aligned to the ask, as they knew it would be the cleanest possible test.”
Once the PSA test was complete for the two large brands, there were still seven other brands to measure. The cost of further PSA testing was prohibitive. This led to a Razorfish recommendation to do a natural experiment.
Natural experiments are those where the investigator does not control all the factors and accepts lower confidence ratings because some factors influencing results are not measured or managed.
To create a suitable control in such an environment, Starwood and Razorfish considered these factors:
- Make it brand agnostic.
- Ensure it reflects online behavior.
- Have it be reflective of overall demand patterns.
The team selected Destination Search (where the keyword is the name of the destination – e.g.: Cape Cod Hotel). This fits the three criteria: It is a more general result (brand agnostic); it is reflective of general online behavior (search); and the practice aligns with the distribution of online engagement with Starwood properties.
“The test was designed to determine if display media drives any incremental booking, or if a user would not have booked without exposure to an ad,” Arun said.
The results were actionable. “We found that campaigns were having very significant impact for some brands,” Michelle said. The brands with lower recognition in the marketplace had higher incrementally (e.g.: the ads were making more of an impact).
“In addition, I felt confident in these Natural Experiment results when they compared similarly to the results of the Sheraton and Westin PSA testing,” she said. Starwood used the results to move media budgets for each brand to a more impactful mix along the buying cycle.
“Natural experiments are not a perfect science, by definition,” Arun cautioned. However, when you do them mindfully and be sure to ‘sniff test’ them with results from a control test, then they can be an important part of the testing program.”
Continual testing is a lot easier today, given the built in tools with many marketing technology solutions from campaign management to email delivery to programmatic display buying. This Starwood commitment illustrates the value of methodical testing to the bottom line – a great way to Stay #TopRight in your marketplace.