101 Things You Should Know

A Report from the eMetrics Analysis Symposium, Summer 2009

 

I wasn’t sure whether to call this:
101 Things You Should Know About Web Analytics
101 Things You Should Know About the Nature of Analysis
or
101 Things You Should Know About Marketing Optimization Analysis I’ll let you decide… 101 Things About Marketing Optimization Analysis.pdf
(about 1.2 MB)

 

They came from all over the world to think about thinking.

The assignment was simple: Address what the human mind brings to the world of online marketing optimization analysis. Disregarding the data itself, the tools used to crunch the numbers and the quality of mathematical models, what is the human element? Prepare an hour long presentation and only present the last ten minutes. Just give us the conclusion.

Fifteen presentations were offered as fodder for two hour-long roundtable discussions. The ideas presented were both familiar and extraordinary, introspective and superficial, outrageous and admirable, and philosophical and pragmatic. There were jokes, metaphors, instructions, pleas, rules and headscratchers. The presenters were corporate practitioners, consultants, vendors, and consulting analysts. The assembled IQ was off the charts and all were entranced and enraptured.

“Cross fertilization of ideas very useful”
“Enlightening and thought-provoking”
“Great insight”
“Stimulating”
“Wow”

Sending Out the Call

After sixteen years of studying online marketing and ten years of investigating online metrics and marketing optimization, I am the founder and producer of a conference called the eMetrics Marketing Optimization Summit. The Summit’s audience had joined forces to create the Web Analytics Association. In its eighth year and transpiring in eight countries, the Summit has become a gathering place for marketers to learn and share knowledge about exploiting online data to improve their websites, optimize their marketing and guide their businesses.

From its start in 2002, the eMetrics Marketing Optimization Summit attracted the best and brightest. They have been intent on shaping and shepherding this nascent industry as it holds incredible value for those able to grasp and capitalize on the potential.

The eMetrics Analysis Symposium of August, 2009 was a chance for those best and brightest to reconvene. One day before hundreds of the uninitiated arrived to get a first-hand look at this brave new world, I invited these subject matter experts to take a fresh look at making the most of an underappreciated data set.

The Presenters

They came from near and far and are a treasure trove of experience, insight and viewpoint. The gathered gurus included:

Gary Angel, President and CTO of Semphonic and creator of the X Change Conference for web analysts with more than 20 years of experience in data analytics for marketing and operations.

Dennis Bradley, Director of Web Analytics, Charles Schwab with 15 years of experience in sales, marketing and analytics.

Vicky Brock, Co-Founder of Highland Business Research specializing in insight and analytics to non-transactional clients such as universities and travel and public sectors and on the Board Directors of the Web Analytics Association.

Jason Burby, Chief Analytics & Optimization Officer, at ZAAZ, one of the largest and most recognized strategic web analytics consulting firms and Co-Chair of the Web Analytics Association Standards Committee. Author of a regular web analytics column on ClickZ and of the highly touted book, Actionable Web Analytics: Using Data to Make Smart Business Decisions

John DeFoe, Vice President of Solution Services at Webtrends where he helps customers take full advantage of the suite of webtrends products for more than eight years.

Bryan Eisenberg, Co-founder and CPO (Chief Persuasion Officer) at FutureNow Inc. where he invented Persuasion Architecture. Bryan is one of the original founders Web Analytics Association and bestselling author of, Call to Action, Waiting for Your Cat to Bark and several other books.

Richard Foley, World Wide Product Manager and Strategist, SAS Institute who served as President and is a Director Emeritus of the Web Analytics Association.

Brian Kelly, CEO at Quantivo has spent 20 years in development and marketing of Analytic Applications and CRM solutions as Director of Business Intelligence Applications at Peoplesoft and large-scale datawarehouse designer at Teradata.

Dylan Lewis, Managing Director at Intuit where he is engrossed in translating online behavior into better website experiences after having been a Senior Consultant at Visual Sciences.

Joe Megibow, VP, Global Analytics and Optimization, Expedia where he deploys leading edge site conversion measurement and site optimization techniques and oversees customer analytics for all of Expedia.com worldwide Joe was a charter employee of Tealeaf Technology and held management positions at Ernst & Young and EDS in their Advanced Technology Group

Angel Morales, Founder of Lightsout Marketing with more than ten years of experience architecting solutions for multi-channel merchants and retailers having led the retail practice for email marketing company ExactTarget.

Bob Page, Head of Analytics Engineering at Yahoo! where he oversees Yahoo!’s Web Analytics and Advertising Analytics after being a co-founder and the CTO of Accrue Software, large-scale web analytics systems.

Judah Phillips, Senior Director, Global Site Analytics at Monster Worldwide where he is responsible for developing the processes, people, and infrastructure for generating actionable insights on user behavior from the web data. Judah blogs on the Web Analytics Demystified website and writes for MediaPost’s Metrics Insider.

Greg Poffenroth, Senior Marketing Optimization Consultant at Ascentium stays tuned to clients’ strategic execution of their online marketing initiatives with an emphasis on the measurement and optimization of various online media like social media, paid & organic.

Aurélie Pols, a world class consultant with Web Analytics Demystified after leading the web analytics practice in Europe for LBi and OX2 for more than eight years. Aurélie is on the Board of Advisors at Next Stage Evolution.

Jennifer Veesenmeyer, VP of Analytics at Stratigent where she specializes in assisting enterprise-level organizations gain executive buy-in, build consensus and facilitate cultural change.

What Was Covered

Some of the following may be considered best practices or tips for aiding with change management. There is advice about critical thinking, some project management tips, a few words of warning and a few fun things to think about. For the most part, this is a list of Things You Need to Know generated by some of the best minds on the subject.

The presentations varied widely and rather than repeat them verbatim, here are the main points from the speakers as well as from the participants during the roundtable discussions. As much as I’d like to attribute the ideas where appropriate (Bob Page and Data Farming!), this is an interpretation rather than a report. So thanks to all the above who participated. Your ideas are valuable and you know who you are!

I recommend reading this once a month and seeing which of the following jump out at you given your circumstances du jour.

Ready?

~~~~~~~~~~~~

eMetrics Analysis Symposium 01

Always think about what people are doing rather than about data. Models and trends and coefficients are great but if it doesn’t relate to how you are conducting business with people, you need to come back to the real world and try again.

eMetrics Analysis Symposium 02

It’s about HOPE

Hypotheses – have an opinion
Observation – understand the human side of the equation
Prediction – make a guess
Experimentation – run some tests

eMetrics Analysis Symposium 03

It is essential that the analysis you do guides a specific course of action. Otherwise you will generate a report that is merely interesting and not useful (valuable).

eMetrics Analysis Symposium 04

Always try to create a story. Those who depend on your insight are not dumb, they just do not have your deep understanding of the data, the circumstances or the implications. If you are smart enough to summarize and explain what you think and why you think that way to a ten year old then you are smart enough to de-jargonify your conclusions to people who look to you for wisdom to make business decisions.

eMetrics Analysis Symposium 05

Every now six months or so, stop all automated reports and see how many recipients notice, how many complain and how many are relieved.

eMetrics Analysis Symposium 06

Be clear about your own biases. We all like to assume everybody sees the world the same way we do and it’s just not so. You are not the target audience so you have to quantify your biases and factor them out of your assessments. “Your opinion, while fascinating, is completely irrelevant.”

eMetrics Analysis Symposium 07

As a corollary to the above, do everything in your power to protect yourself from arbitrary decisions made from on high. This problem of management by Hippo (HIghest Paid Person’s Opinion) has been with us since time immemorial but is especially prevalent when dealing with online matters. Everybody is an expert and the most senior person in the room is likely to start some judgmental travesty with, “Well, when I surf the web…” Always use data to protect yourself from them and from your own biases.

eMetrics Analysis Symposium 08

Analysis is for those with business acumen. Otherwise you are a statistician – a user of tools rather than a diviner of understanding. Where the division of labor happens in your company varies.

eMetrics Analysis Symposium 09

Analysis should be prioritized when

Revenue is at risk
The boss’s boss’s boss asks for it
It does not overwhelm the department
It requires an analyst rather than simply being self-served

eMetrics Analysis Symposium 10

Be careful – all statistical results are political.

eMetrics Analysis Symposium 11

Never use analytics in anger.

eMetrics Analysis Symposium 12

A great analyst has more white matter than most. Grey mater are the neurons that can solve a linear problem. Lots of grey matter is important. White matter is the stuff that connects the linear processors in a non-linear way and causes flashes of insight into why things happen instead of just how they happened.

“That doesn’t look right… ”
“That reminds me of something…”
“I wonder if that acts like this other correlation…”

eMetrics Analysis Symposium 13

Analysts are:

eMetrics Analysis Symposium 14

A good analyst is always on the lookout for anomalies. What stands out? What has changed the most?

eMetrics Analysis Symposium 15

All the data crunching tools in the world won’t help you as much as the ability to draw conclusions. Practice your skills. Start with Venn Diagrams.

eMetrics Analysis Symposium 16

The only metrics that matter are money and customer satisfaction

eMetrics Analysis Symposium 17

Understand that there are different kinds of problem solving. Read “Analytical Thinking: How To Take Thinking Apart And What To Look For When You Do” criticalthinking.org

eMetrics Analysis Symposium 18

Develop multiple ways of thinking about things.

Chronologically
From the end result backwards
Analogies
Spatially
From the customers’ perspective
Upside down

eMetrics Analysis Symposium 19

Insights are most valuable if they help an individual get recognition, a raise and/or a promotion. Otherwise, that insight might never be put to use.

eMetrics Analysis Symposium 20

Create an Insight Portal like Yahoo!’s. Any analyst (or anybody) can post an insight, uploading the report on which they based their opinion with links to the data. These posts are tagged, searchable, rated, commented on and the number of times they are viewed (and by whom) noted. They include conference trip notes, PowerPoint decks, etc.

eMetrics Analysis Symposium 21

All too often, business people approach analysts after the fact and ask, “What happened?” Once you train them to bring you in at the beginning of a project, you can start with The Dance. During The Dance, your job is to understand how much they know and how much they are going to be able to understand. How much experience do they have with the data, data analysis and the application of results to a business issue? You must also determine their definition of success for the project and what decisions are going to be made based on the results. Taken to its logical extreme, you can get them to pre-define their actions depending on the results.

If this KPI goes up by 25% we’ll do X
If this KPI goes down by 15% we’ll do Y
If this KPI does not change we’ll do Z

eMetrics Analysis Symposium 22

“Decisioneering” – coined by Chris Worland from Microsoft

eMetrics Analysis Symposium 23

Offering to improve business results will raise the profile of the analyst’s domain.

We can improve
this number
by this percent
in this time frame
if we run this test
for which we will need these resources

eMetrics Analysis Symposium 24

There is a danger of diminishing returns when going after perfect data. It is not perfect and never will be. Analysts who spend too much time cleansing their data or continually cannot make a recommendation because there is not enough data should be retrained or replaced.

eMetrics Analysis Symposium 25

Analysis is about communication. Be sure to create reports that are accessible to the target audience. The New York Times got much more attention when they formatted their charts to look like New York Times financial page reports.

eMetrics Analysis Symposium 26

Define the problem. Given an hour to save the world, Einstein said he would spend 59 minutes defining the problem and one minute solving it.

eMetrics Analysis Symposium 27

Act like a recommendation engine. “You know, people who liked this report also liked this one.”

eMetrics Analysis Symposium 28

Never automate report distribution. It is an invitation to ignore the data. If they are not using the information to make business decision, they do not need to know the most visited page or the most popular exit page.

eMetrics Analysis Symposium 29

If you are going to highlight key learnings or keen insights, keep it to a minimum. Talk about two or three things at once – at most. The rest can wait for another conversation.

eMetrics Analysis Symposium 30

If you can match the behavior data with the attitudinal data, that’s good. If you can match the two of those to verbatim comments, that’s magic. People don’t respond as well to, “We have a 12% non-completion rate,” as they do to, “57 people told us they got stuck trying to change their profile settings and then we saw more than 750 were unsuccessful yesterday. We should do something.”

eMetrics Analysis Symposium 31

If you can figure out what report will help make your boss’s boss look good by making better decisions, you have gone from being valuable to being indispensible.

eMetrics Analysis Symposium 32

We started our doing analytics manufacturing. We are learning how to do analytics mining. Is it time to try our hand at analytics farming?

Analytics Manufacturing is all about the acquisition, refinement, assembly and maintenance of analytics as a continuous enterprise. Teams of specialists implement acquisition procedures and management strategies with a focus on production of large stocks of analytics with interchangeable labor. Yes, you end up with continuity of process which is easier to teach (and learn) and you gain some productivity. But this is a world of report monkeys – interchangeable skilled labor doing repetitive leg work.

Analytics Mining sees analysis as a valuable resource with insights to be extracted and refined. It requires solid tools to quickly extract value from a sea of data and rewards the individual analyst out sifting for gold nuggets. It’s easier to seize opportunity and act quickly. While manufacturing works with a “datawarehouse,” data mining sifts through that store of ore in search of the correlation that can enlighten the organization.

Analytics Farming looks at analytics as a continuing collaborative activity. It is performed by groups working together on changing individual and common goals and focuses on continuous cultivation and community. It’s a social activity where discovery and success are shared.

This communal approach puts the emphasis on the big picture and the meta understanding of the entire environment. Rather than raw material going in and finished goods coming out the other end, this is a delicate dance with the environment and with others in the organization.

More than just formulae and tabulations, analytics farming works in an ecosystem and a longer-term view. If the seeds (page tags) are properly planted, then the harvest (behavioral data set) can be reaped in a timely way, processed into usable ingredients and baked into a pleasing loaf of bread (valuable reports). If the bread is stored and distributed properly, it can be consumed as nourishment.

The analogy can drop all the way down to soil conditions (corporate culture), fertilizer (management sponsorship), rainfall (sufficient funding) and etc. The concept is that this is less a mechanistic or opportunistic endeavor, but a long-term it-takes-a-village enterprise.

To be successful at Analytics Farming, you have to foster an environment of collaboration, be willing to change with the weather, and accept that sometimes the crops are abundant and sometimes they fail.

eMetrics Analysis Symposium 33

Never hit people with a 16 ton sack of data. Make it palatable. Make it bite-sized. Don’t go too deep. Don’t baffle them with the reasons for your conclusions / hypotheses / recommendations.

Think of data like food. Too little leads to malnutrition. Too much leads to indigestion. Too bland is boring. Too rich leads to a mad-dash to the bathroom.

eMetrics Analysis Symposium 34

Always deliver context. A measurement by itself (I am 6 feet tall) isn’t even interesting. A measurement in context (I am 6 feet tall and trying to enjoy a tea party in a tree-house built for 8 year-olds) is interesting and leads to several trains of thought.

eMetrics Analysis Symposium 35

Host an event at your organization where you can teach analysis. Start non-threateningly with a brown bag lunch or an internal Web Analytics Wednesday and move up to an Analysis Boot Camp. Boot Camp should include a diversity of disciplines like user experience design, marketing strategy, business analysis, search, creative, and development.

eMetrics Analysis Symposium 36

Spend some time with a 4-year-old to truly grasp the meaning and the many uses of the question, “Why?”

eMetrics Analysis Symposium 37

Heed Descartes’ 4 principles:

  1. Accept only that input of which you are sure
  2. Divide a complex problem into as small parts as necessary
  3. Solve the simplest problems first
  4. Use induction and deduction to make as accurate and complete an inventory of the data as possible

eMetrics Analysis Symposium 38

Present yourself as an independent consultant who can look at things in a different way, from a different angle or in a different light. If you are an insider, make use of consultants – different consultants – from time to time to shake up your point of view.

eMetrics Analysis Symposium 39

A good analyst understands the business and knows whether the issue at hand is really about saving or making money this time.

eMetrics Analysis Symposium 40

Analysis is not a once-and-done endeavor. Times change, economies change, popular culture changes, competitors change. So you must embrace analysis as an on-going effort – a continuous process.

eMetrics Analysis Symposium 41

Understand the compensation packages of the people to whom you are delivering the analysis. To understand their motivation is to understand why they care about some facts and correlations not about others.

eMetrics Analysis Symposium 42

Market and sell the idea of analytics internally. Spread the enthusiasm as you would your favorite candy. Describe it’s goodness. Explain that it is worth the price. Offer a taste. Get them hooked. Send out small samples in personal emails. See who responds with the “That’s interesting!” look in their eyes. If you don’t see it there, back away slowly.

eMetrics Analysis Symposium 43

Market and sell your team internally. Put up profile pages. Show what they’ve done before coming to your organization and write stories about how they have helped others inside your organization. Explain what drives each of them and what types of problems they like to solve.

eMetrics Analysis Symposium 44

Don’t get tied in knots about not having enough data or having data that is clean enough. You never will and it never is. Perfect is the enemy of good enough. So while you’re waiting for better data:

Go after quick wins
Find ways to make others look good
Review the basics like search phrases and exit pages
Compare and contrast against other data sources like voice of customer

eMetrics Analysis Symposium 45

Good decisions are based on a logical process of considering the information at hand. A good decision does not mean the decision is right. Proper decision-making process that lacks insight, intuition and understanding is an algorithm. The human aspect is being able to consider things that are not necessarily part of the input set – like the weather.

eMetrics Analysis Symposium 46

The roles are often combined, but it’s valuable to remember that there are people who are better at some things than others.

Data capture
Data cleansing
Data integration
Insight realization
Insight distribution
Insight application
Problem definition
Problem solving
Solution application

eMetrics Analysis Symposium 47

Data, by themselves, are not actionable. Our tools are there to help us gather and manage the data but do not help identify the best way forward. They are not able to recognize and suggest a new, potentially better action to take. We must use our tools to aid in the understanding of the situation, but cannot expect those tools to help in decision making.

eMetrics Analysis Symposium 48

It is presumptuous to assume we can understand how the factors that actually matter in complex systems are related to each other and how changes in one impact another. The interconnectedness of elements and the law of unintentional consequences must always be kept in mind lest we imagine that we have infinite control.

eMetrics Analysis Symposium 49

Creating understanding is all about finding the right level of simplification – for your audience – to describe a system.

eMetrics Analysis Symposium 50

Are you reviewing and reporting on what your competitors are looking at on your website?

eMetrics Analysis Symposium 51

Deliver research and analysis that fits the current business process. They will not listen if the first thing you tell them is that they are doing everything wrong.

eMetrics Analysis Symposium 52

Centralize your data crunching and analysis at up front in order to get it organized. Next, decentralize as different business units understand the value and learn how. Finally, re-centralize in order to glean best practices and re-standardize across all departments. Repeat.

eMetrics Analysis Symposium 53

Look at the data, formulate a hypothesis, construct a model and then put it away for a day. When you come back to it, you will realize there are a few problems and will see four new possibilities.

eMetrics Analysis Symposium 54

When asking about business goals, do not settle for “raise revenue” or “cut costs”. Demand to know by how much and how soon. Now you will have a specific problem you can solve and the metrics will reveal themselves.

eMetrics Analysis Symposium 55

Get to know three key people:

  1. The person who controls the budget – they control all
  2. The person who decides how to achieve the goals – they need your help
  3. The person responsible for the outcomes – they will come to depend on you

eMetrics Analysis Symposium 56

Be willing to go on fishing expeditions. A total focus on outcomes is good, but exploration yields value as well.

eMetrics Analysis Symposium 57

How does your audience digest information? Some people like spreadsheets, some like charts, some like graphs and some like stories. Create your reports so they appeal to a variety of sense. The people inside your organization are your customers so segment them as to their learning style – then cater to it.

eMetrics Analysis Symposium 58

You don’t need to be precise, you only need to be compelling.

eMetrics Analysis Symposium 59

Visualization is a very persuasive tool which can more quickly illustrate relationships and changes over time. Periodically publish the most compelling images in a way that people can share. A picture is worth a thousand words and an animation is worth a thousand pictures.

eMetrics Analysis Symposium 60

Be a change agent. Read up on change management so you don’t risk too much but aren’t guilty of the sin of sitting on your hands.

eMetrics Analysis Symposium 61

A recommendation engine can be used internally as well. “People who liked this report, also liked that one.” Social rating is another form of recommendation. “207 people rated this report five stars.”

eMetrics Analysis Symposium 62

We aren’t decision makers, we are “decision enablers.” We analyze customer interactions with our websites, provide hypotheses about the world around us, deliver insights and suggest experiments. Our job is to help others make decisions by showing them the alternatives and suggestion which can be expected to have the best outcome.

eMetrics Analysis Symposium 63

Create a training and learning environment starting with interns and give people a clear, steady path to mastermind. Have a readily available chart of skills and required knowledge so people know what their career path looks like.

eMetrics Analysis Symposium 64

Do you know what tools you have available? Every four to six months, do a technology and data inventory. Other departments may have licensed a tool you haven’t seen before or are using a free tool. Some may be collecting a different type of data that might be enormously valuable when correlated with what you know you have.

eMetrics Analysis Symposium 65

Change all of your reports from the technical and tactical to the business and financial. Everybody in the company will understand a report about revenue & expense and profit & loss while only a few will follow a conversation about unique visitors vs. cookies.

eMetrics Analysis Symposium 66

When trying to effect change with data, use facts that are

Fun – something to get their attention
Interesting – something to capture their imagination
Useful – something to secure their enthusiasm

eMetrics Analysis Symposium 67

Go buy lunch for somebody in the finance department. Today. Just get to know them, understand their motivation and their perspective. Learn their language. Earn their trust.

eMetrics Analysis Symposium 68

Statistics are lumber – you get more out of them if you build something than if you hit people over the head with a 2×4.

eMetrics Analysis Symposium 69

“The most exciting phrase to hear in science, the one that heralds new discoveries, is not Eureka! (I found it!) but rather, “Hmmm…. that’s funny….”
– Isaac Asimov

eMetrics Analysis Symposium 70

Financial reports show impacts to transactions, but give little indication of where issues were occurring. Web analytic tools aggregate site usage stats but lack detailed customer experience data and information on customer intent. System logs and IT site monitors do not track details of customer experience issues. Code logging requires meticulous development efforts to identify and record every fork that could be exercised and it creates performance issues. Customer feedback mechanisms are not adequate and monitored or acted upon. Email is untargeted and drive spotty results. All of these tools are useless until we can integrate and correlate the data from them and then take some sort of action.

eMetrics Analysis Symposium 71

Until you have sufficient resources for the big datawarehouse in the sky, you can integrate data streams at the dashboard level and train people to do the pattern matching and critical thinking.

eMetrics Analysis Symposium 72

Analysis

Begins with investigation
Leads to the uncovering of clues
Extracts meaning from clues
Assembles clues into a deduction
Enables the crafting of a story that narrates your investigation and final deductions
Generates a presentation, report, and additional docs that include recommendations

eMetrics Analysis Symposium 73

Your opinion though interesting is completely irrelevant. You are not the target audience.

eMetrics Analysis Symposium 74

Collect data for customers rather than about them. Use analysis to improve individual customer experience (targeting) and you will be rewarded. Use analysis sell more stuff and you will be frustrated.

eMetrics Analysis Symposium 75

First, look for anomalies.
Next, look for changes.
Then, look for patterns.

eMetrics Analysis Symposium 76

The more valuable segments tend to be small. Dividing visitors into male and female is helpful but the real gold is found in that fraction who share a half a dozen characteristics or behaviors.

eMetrics Analysis Symposium 77

When it comes to change management, do not preach to the choir – preach to the almost converted. Sing to those who are already humming.

eMetrics Analysis Symposium 78

Know what you don’t know.

This is what we know.
This is what we think.
This is the test we can run to find out.
Here’s what we learned from the test.
This is what we still don’t know.

eMetrics Analysis Symposium 79

Make a point of reaching out to the advertising people/department/agency at your organization and understand their process, their limitations and their motivations.

eMetrics Analysis Symposium 80

Measure the success of your measurement. Find a way to monitor how much people are using marketing analysis information to make business decisions. Prove that those who follow the results make more improvements than those that fly blind.

eMetrics Analysis Symposium 81

Prioritize the analysis you do depending on

Whether revenue is at risk
Who wants to know
How hard it is to answer the question
Whether it really requires an analyst or can be self-served How soon it’s needed
The purpose of the request

eMetrics Analysis Symposium 82

Occasionally take some time to ruminate about the data. Stir it with a stick and see if anything interesting floats to the surface. Look for patterns and correlations in the spirit of discovery. Sometimes the most interesting things come up when you’re just looking at the data rather than trying to solve a problem. But beware – this is dessert, not the main course. You must solve specific problems in order to earn the time to ponder.

eMetrics Analysis Symposium 83

Spend a day in the contact center. Take some calls about your website, talk to the service representatives and come to understand who your customers really are. Then you will be able to think of ways to integrate what data there is in the contact center with the data you already have.

eMetrics Analysis Symposium 84

Be ever vigilant against misoneism: a hatred or fear of change or innovation.

eMetrics Analysis Symposium 85

Most people use facts to confirm their beliefs. It’s human nature. They look for the data that supports their hypotheses. You are a trained analyst and have learned to guard against this practice but your internal customers have not. Those who come to you for answers may be looking for specific answers and will blame you for not doing your job if the actual data do not corroborate their desired results. You must teach them without insulting or alienating them.

eMetrics Analysis Symposium 86

“It is of the highest importance in the art of detection to be able to recognize, out of a number of facts, which are incidental and which vital. Otherwise your energy and attention must be dissipated instead of being concentrated.”
– Sherlock Holmes

It may not matter how often the customer has purchased before or it may be the most significant attribute of all. Finding quick ways to determine the relative weight of factors will help you cut through the noise faster.

eMetrics Analysis Symposium 87

When trying to understand a problem, ask for clarification. Ask the person on the other side of the table to elaborate. Ask for an example. Ask for the opposite of their situation. Ask them to show you what success looks like. Ask them to be more specific. Ask I they already have an opinion. Ask them how they will use the information they seek. What decisions will be made using the resulting analysis?

eMetrics Analysis Symposium 88

The number of insights is not proportional to the amount of data. There is no correlation.

eMetrics Analysis Symposium 89

Your job is part marketing, part IT, part finance, part market research, part customer service, and etc. That means you are a service organization and are only valuable you truly understand what each of those departments does for a living.

eMetrics Analysis Symposium 90

Etiology (alternatively aetiology, aitiology) is the study of causation, or origination. The word is derived from the Greek aitiologia, “giving a reason for” (aitia, “cause”; and -logia).

The word is most commonly used in medical and philosophical theories, where it is used to refer to the study of why things occur, or even the reasons behind the way that things act, and is used in philosophy, physics, psychology, government, medicine, theology and biology in reference to the causes of various phenomena. An etiological myth is a myth intended to explain a name or create a mythic history for a place or family.
Wikipedia

eMetrics Analysis Symposium 91

“I don’t know – let’s run a test,” is often the most insightful thing you can say.

eMetrics Analysis Symposium 92

“Today’s scientists have substituted mathematics for experiments, and they wander off through equation after equation, and eventually build a structure which has no relation to reality.”
– Nikola Tesla

eMetrics Analysis Symposium 93

Analysis is not sequential at first. It starts with making unobvious connections and chasing hunches. It cognition, intuition and ingenuity. Then we can start on the path of collecting data, reviewing it and testing assumptions and hypotheses.

eMetrics Analysis Symposium 94

A good analyst always asks what problem they are trying to solve. And then asks the question behind the question. If they want to know the best way out of the building, it depends on why they want to leave. If they want a taxi to the airport, the front door is best. If they want to drive to the airport, the door to the parking lot or garage is best. If they want to take out the garbage, the back door is best. If the building is on fire, the solution is very, very different.

eMetrics Analysis Symposium 95

Think of data as evidence. There is inconsequential evidence, distracting evidence, circumstantial evidence, etc. There is sometimes the need to know something beyond a shadow of a doubt and sometimes one only requires a preponderance of evidence. Evidence should be respected – one must always be careful not to contaminate the evidence.

eMetrics Analysis Symposium 96

“I do not paint things. I paint only the differences between things.”
– Henri Matisse

eMetrics Analysis Symposium 97

Just like web visitors, people who consume your reports and view your dashboards cannot be bothered to scroll.
Ever.

eMetrics Analysis Symposium 98

No matter how many people you have and how talented they are, periodically bring in an outside consultant. They come with a different perspective, different experience and are willing to ask those critical, random questions that scare everybody on your team.

eMetrics Analysis Symposium 99

Business acumen comes from having sufficient knowledge about organizational goals and what everybody else in the company does in order to use your superpowers to help them achieve their goals. Simply providing reports to people only gives them the ability to ask questions in just the right way to cover their own backsides.

eMetrics Analysis Symposium 100

Hire people who like puzzles, detective novels, science fiction, crosswords and whose eyes light up when they learn something new.

eMetrics Analysis Symposium 101

When it comes to segmenting customers by behavior:

  1. Some do and some don’t.
  2. The differences aren’t that great.
  3. It’s more complicated than that.

– Bernard Berelson
Human Behavior: An Inventory of Scientific Findings (1964)


What follows are a few things I’ve picked up since the eMetrics Analysis Symposium. I hope you find them useful.


eMetrics Analysis Symposium 102

A website without metrics is just a hobby.
– Peter Andersen, Danske Bank

eMetrics Analysis Symposium 103

We are using analytics as a treasure map to fund our search for truth.
– Matt Cutler & Brett Crosby, X Change 2009

eMetrics Analysis Symposium 104

We are not accountants, we are statisticians. Statistics means never having to say you’re certain.
– Bob Page, X Change 2009

eMetrics Analysis Symposium 105

The truth changes depending on where the tag is on the page
– John Pastana, X Change 2009

Stay tuned for more…