We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person's Opinion (HiPPO). We review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We focus on several areas that are critical to experimentation, including statistical power, sample size, and techniques for variance reduction. We describe common architectures for experimentation systems and analyze their advantages and disadvantages. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments.
The talk is based on an article in KDD 2007
Tuesday - 11:15 am - Diplomat Room
The environment web analytics exists in is becoming more complex. It's not just about web pages any more, or even about the web site. You have new technology platforms on the site, new sources of traffic, and multi-channel concerns to deal with. How do you sort this all out, report on it in an actionable way, and still make sure people understand what the heck you are talking about? And what are the implications of all this for managers and the culture of analytics? Find out about this major new opportunity to get management's attention and two specific ideas for taking advantage of it.
Tuesday - 1:30 pm - Diplomat Room
With an even-larger percentage of advertising investment going online, it's critical to maximize ROI at every stage in the process. Huge opportunities exist to optimize the final component of any successful campaign - the landing page - which closes the deal. Robert will describe how The Weather Channel improved conversions by up to 225% using multivariate optimization.
Tuesday - 4:00 pm - Diplomat Room
With more than 15 years of experience to draw from in market research and analysis, Nick will explain how and why VistaPrint implements this kind of testing and will outline the success that the company has enjoyed as a result. Learn about maximizing promotions and merchandising for bottom line profit, how the utilization of the cross sell and up-sell technologies maximize conversion using mega split runs to increase contribution margin. Find out how to run more than 20 tests at one time to gather data on over 2,000 user experiences. Plus, learn about price elasticity, site traffic, and much more.
Tuesday - 4:00 pm - Diplomat Room
From the basics to the advanced aspects of behavioral targeting difference, Anil will explore how consumers, publishers and advertisers can use and benefit from BT. From the available technologies to the tried and true techniques to the preponderance of privacy pains. He will also show how you can use your exiting web analytics tools to deliver Behaviorally Targeted content, messages and products to drive higher conversions.
Tuesday - 5:00 pm - Diplomat Room
Back by popular demand, Seth reviews how Microsoft uses customer profiles and anonymous customer information to create targeted web experiences...real time. The goal is to deliver content that's timely and relevant to the customer improves content quality, visitor satisfaction, and desired outcomes. Seth will describe customer experience scenarios to illustrate the value of actively engaging customers throughout their visit to the site. From philosophy to technology, Seth brings his extensive years of marketing and data management experience to bear on making the most of online customer intelligence.
Wednesday - 11:10 am - Hampton Room
As one of the largest worldwide merchandisers of more than 70,000 properties worldwide, hotels.com is enjoying solid growth. Using a combination of classical web metrics, customer feedback, performance monitoring, and even the ability to playback individual visits, the company continues to improve the online experience and optimize its investment. Joe will share how it's done, what it takes and who you need on your team to make a go of it.
Wednesday - 1:30 pm - Hampton Room
Behavioral Targeting has been reinvented in Europe. A stock exchange for profiles is being established and predictive behavioral targeting redefines the way advertisers can talk to their traditional target groups. Targeting including milieus, demographics and offline consuming behavior without facing a decrease in reach. Oliver will talk about advancements that have been made in the past 12 months in Europe and give detailed insight into methods used and players on the European market.
Wednesday - 2:30 pm - Hampton Room