Live, from the Google summit in California

Web Optimiser Day opened with Dan Siroker, previously of Google and more recently of the Obama Campaign New Media Team.

Google’s Web Optimiser is a free tool that facilitates testing different combinations of content in a live environment. The goal is to deduce which combination of content best drives conversions.

Dan shared lessons he learned about Web Optimiser testing in the trenches of the election campaign, and illustrated how these could be applied to commercial Web optimisation projects.

In November 2007 Barack Obama visited Google:

On that day Barack declared himself to be a believer in evidence, science, facts and feedback. He impressed all around by fielding Eric Shmidt’s standard Google interview question on how to best sort a million 32 bit integers. He said,

“I think the bubble sort would be the wrong way to go.”

Like all good politicians, Barack didn’t actually answer the question, but he did give an impressively intelligent response (a bubble sort would solve this problem, but the sort would run in O(n^2) time making it sub optimal).

Dan was inspired to join the Obama campaign and recruited friends, family and many Computer Science PhDs, and launched an impressive new media strategy, which included numerous rounds of Web Optimiser testing. He shared some lessons he learned with us.

Lesson 1: Define Quantitative Success Metrics

In the case of the election campaign, the over-riding success would be the inauguration of Barack Obama.

The success metrics however, need to be more granular in order to be useful. For example, to measure the success of an Adwords campaign, the first relevent metric is the CPC for generating visits to the campaign site.

The subsequent ‘sign up rate’ and the ‘donation per recipient measured in dollars’ complete the cycle.

Comparing campaigns based on these metrics will reveal the success (or otherwise) of a particular campaign.

Dan’s response to a question on how he fed his findings to the wider campaign team was comforting: he did not make use of KPI sheets because he knew that they would not be read.  It is a lesson in itself that the Web Analyst should go full speed ahead — I do believe in educating the wider organisation, but in order to progress, it is often necesarry to just take the reins and provide results.  Dan first proved the worth of testing and then he was given the authority to make changes to the site based on his findings.

Lesson 2: Question Assumptions

The Web Optimiser takes the guess work out of design. Decisions are based on evidence, as opposed to opinion. For example:

Having landed on the Splash Page, which of the following button text do you think would work best at encouraging visitors to engage with the site?

• Sign Up
• Learn More
• Join Us Now
• Sign Up Now

The consensus at Google was that ‘Join Us Now’ would work best, however Dans testing with the Web Optimiser revealed that the ‘Learn More’ button drove the most conversions.

Lesson 3: Divide and Conquer

Moving beyond straight forward testing, Dan then considered whether different combinations of content would work better for different segments of the audience.

For example, the audience for the campaign could be divided into 3 different groups:

• those that had not yet signed up
• those that had signed up but not donated
• those that had signed up and donated already

Combining the Web Optimiser tool, cookies and on page javascript, he tested content on these audience segments separately.  Different button text worked best for different segments of the audience and were deployed as such.

• Donate Now
• Please Donate
• Why Donate
• Donate and Get a Gift
• Contribute

Lesson 4: Don’t reinvent the wheel

Dan told us that there are many open source tools available and we should use them, instead of trying to develop in-house tools ourselves.

In particular, Open Report allowed Dan to automate ad hoc reports, which freed up his team’s time for optimisation and testing.

Lesson 5: Take Advantage of circumstance

Obama’s campaign site quickly responded to offline events that were driving visits to the site, keeping content fresh and capitalising on these visits.

For example, in the 48 hours following Sarah Palins somewhat scary speech at the Republican National Convention in September 2008, $10 million was donated to the Obama campaign within 48 hours.

The following video outlines how Dan used particularly lessons 3 and 5 in order to optimise the campaign site on voting day, by segmenting geographically as well as attitudinally.

Dan is currently developing www.carrotsticks.com, a site that helps kids learn maths; it’s worth a look.

Today is the first of two Google Analytics days at the GAAC summit, and Clodagh and I will also be attending Urchin day on Friday…. assuming we survive trampoline dodge ball tonight.