A Report from the eMetrics Analysis Symposium, Summer 2009
I wasn’t sure whether to call this:
101 Things You Should Know About Web Analytics
101 Things You Should Know About the Nature of Analysis
101 Things You Should Know About Marketing Optimization Analysis I’ll let you decide… 101 Things About Marketing Optimization Analysis.pdf
(about 1.2 MB)
They came from all over the world to think about thinking.
The assignment was simple: Address what the human mind brings to the world of online marketing optimization analysis. Disregarding the data itself, the tools used to crunch the numbers and the quality of mathematical models, what is the human element? Prepare an hour long presentation and only present the last ten minutes. Just give us the conclusion.
Fifteen presentations were offered as fodder for two hour-long roundtable discussions. The ideas presented were both familiar and extraordinary, introspective and superficial, outrageous and admirable, and philosophical and pragmatic. There were jokes, metaphors, instructions, pleas, rules and headscratchers. The presenters were corporate practitioners, consultants, vendors, and consulting analysts. The assembled IQ was off the charts and all were entranced and enraptured.
“Cross fertilization of ideas very useful”
“Enlightening and thought-provoking”
Sending Out the Call
After sixteen years of studying online marketing and ten years of investigating online metrics and marketing optimization, I am the founder and producer of a conference called the eMetrics Marketing Optimization Summit. The Summit’s audience had joined forces to create the Web Analytics Association. In its eighth year and transpiring in eight countries, the Summit has become a gathering place for marketers to learn and share knowledge about exploiting online data to improve their websites, optimize their marketing and guide their businesses.
From its start in 2002, the eMetrics Marketing Optimization Summit attracted the best and brightest. They have been intent on shaping and shepherding this nascent industry as it holds incredible value for those able to grasp and capitalize on the potential.
The eMetrics Analysis Symposium of August, 2009 was a chance for those best and brightest to reconvene. One day before hundreds of the uninitiated arrived to get a first-hand look at this brave new world, I invited these subject matter experts to take a fresh look at making the most of an underappreciated data set.
They came from near and far and are a treasure trove of experience, insight and viewpoint. The gathered gurus included:
Gary Angel, President and CTO of Semphonic and creator of the X Change Conference for web analysts with more than 20 years of experience in data analytics for marketing and operations.
Dennis Bradley, Director of Web Analytics, Charles Schwab with 15 years of experience in sales, marketing and analytics.
Vicky Brock, Co-Founder of Highland Business Research specializing in insight and analytics to non-transactional clients such as universities and travel and public sectors and on the Board Directors of the Web Analytics Association.
Jason Burby, Chief Analytics & Optimization Officer, at ZAAZ, one of the largest and most recognized strategic web analytics consulting firms and Co-Chair of the Web Analytics Association Standards Committee. Author of a regular web analytics column on ClickZ and of the highly touted book, Actionable Web Analytics: Using Data to Make Smart Business Decisions
John DeFoe, Vice President of Solution Services at Webtrends where he helps customers take full advantage of the suite of webtrends products for more than eight years.
Bryan Eisenberg, Co-founder and CPO (Chief Persuasion Officer) at FutureNow Inc. where he invented Persuasion Architecture. Bryan is one of the original founders Web Analytics Association and bestselling author of, Call to Action, Waiting for Your Cat to Bark and several other books.
Richard Foley, World Wide Product Manager and Strategist, SAS Institute who served as President and is a Director Emeritus of the Web Analytics Association.
Brian Kelly, CEO at Quantivo has spent 20 years in development and marketing of Analytic Applications and CRM solutions as Director of Business Intelligence Applications at Peoplesoft and large-scale datawarehouse designer at Teradata.
Dylan Lewis, Managing Director at Intuit where he is engrossed in translating online behavior into better website experiences after having been a Senior Consultant at Visual Sciences.
Joe Megibow, VP, Global Analytics and Optimization, Expedia where he deploys leading edge site conversion measurement and site optimization techniques and oversees customer analytics for all of Expedia.com worldwide Joe was a charter employee of Tealeaf Technology and held management positions at Ernst & Young and EDS in their Advanced Technology Group
Angel Morales, Founder of Lightsout Marketing with more than ten years of experience architecting solutions for multi-channel merchants and retailers having led the retail practice for email marketing company ExactTarget.
Bob Page, Head of Analytics Engineering at Yahoo! where he oversees Yahoo!’s Web Analytics and Advertising Analytics after being a co-founder and the CTO of Accrue Software, large-scale web analytics systems.
Judah Phillips, Senior Director, Global Site Analytics at Monster Worldwide where he is responsible for developing the processes, people, and infrastructure for generating actionable insights on user behavior from the web data. Judah blogs on the Web Analytics Demystified website and writes for MediaPost’s Metrics Insider.
Greg Poffenroth, Senior Marketing Optimization Consultant at Ascentium stays tuned to clients’ strategic execution of their online marketing initiatives with an emphasis on the measurement and optimization of various online media like social media, paid & organic.
Aurélie Pols, a world class consultant with Web Analytics Demystified after leading the web analytics practice in Europe for LBi and OX2 for more than eight years. Aurélie is on the Board of Advisors at Next Stage Evolution.
Jennifer Veesenmeyer, VP of Analytics at Stratigent where she specializes in assisting enterprise-level organizations gain executive buy-in, build consensus and facilitate cultural change.
What Was Covered
Some of the following may be considered best practices or tips for aiding with change management. There is advice about critical thinking, some project management tips, a few words of warning and a few fun things to think about. For the most part, this is a list of Things You Need to Know generated by some of the best minds on the subject.
The presentations varied widely and rather than repeat them verbatim, here are the main points from the speakers as well as from the participants during the roundtable discussions. As much as I’d like to attribute the ideas where appropriate (Bob Page and Data Farming!), this is an interpretation rather than a report. So thanks to all the above who participated. Your ideas are valuable and you know who you are!
I recommend reading this once a month and seeing which of the following jump out at you given your circumstances du jour.
Always think about what people are doing rather than about data. Models and trends and coefficients are great but if it doesn’t relate to how you are conducting business with people, you need to come back to the real world and try again.
It’s about HOPE
Hypotheses – have an opinion
Observation – understand the human side of the equation
Prediction – make a guess
Experimentation – run some tests
It is essential that the analysis you do guides a specific course of action. Otherwise you will generate a report that is merely interesting and not useful (valuable).
Always try to create a story. Those who depend on your insight are not dumb, they just do not have your deep understanding of the data, the circumstances or the implications. If you are smart enough to summarize and explain what you think and why you think that way to a ten year old then you are smart enough to de-jargonify your conclusions to people who look to you for wisdom to make business decisions.
Every now six months or so, stop all automated reports and see how many recipients notice, how many complain and how many are relieved.
Be clear about your own biases. We all like to assume everybody sees the world the same way we do and it’s just not so. You are not the target audience so you have to quantify your biases and factor them out of your assessments. “Your opinion, while fascinating, is completely irrelevant.”
As a corollary to the above, do everything in your power to protect yourself from arbitrary decisions made from on high. This problem of management by Hippo (HIghest Paid Person’s Opinion) has been with us since time immemorial but is especially prevalent when dealing with online matters. Everybody is an expert and the most senior person in the room is likely to start some judgmental travesty with, “Well, when I surf the web…” Always use data to protect yourself from them and from your own biases.
Analysis is for those with business acumen. Otherwise you are a statistician – a user of tools rather than a diviner of understanding. Where the division of labor happens in your company varies.
Analysis should be prioritized when
Revenue is at risk
The boss’s boss’s boss asks for it
It does not overwhelm the department
It requires an analyst rather than simply being self-served
Be careful – all statistical results are political.
Never use analytics in anger.
A great analyst has more white matter than most. Grey mater are the neurons that can solve a linear problem. Lots of grey matter is important. White matter is the stuff that connects the linear processors in a non-linear way and causes flashes of insight into why things happen instead of just how they happened.
“That doesn’t look right… ”
“That reminds me of something…”
“I wonder if that acts like this other correlation…”
- Pattern recognizers
- Idea generators
- Quick learners
- Technical (sort of)
A good analyst is always on the lookout for anomalies. What stands out? What has changed the most?
All the data crunching tools in the world won’t help you as much as the ability to draw conclusions. Practice your skills. Start with Venn Diagrams.
The only metrics that matter are money and customer satisfaction
Understand that there are different kinds of problem solving. Read “Analytical Thinking: How To Take Thinking Apart And What To Look For When You Do” criticalthinking.org
Develop multiple ways of thinking about things.
From the end result backwards
From the customers’ perspective
Insights are most valuable if they help an individual get recognition, a raise and/or a promotion. Otherwise, that insight might never be put to use.
Create an Insight Portal like Yahoo!’s. Any analyst (or anybody) can post an insight, uploading the report on which they based their opinion with links to the data. These posts are tagged, searchable, rated, commented on and the number of times they are viewed (and by whom) noted. They include conference trip notes, PowerPoint decks, etc.
All too often, business people approach analysts after the fact and ask, “What happened?” Once you train them to bring you in at the beginning of a project, you can start with The Dance. During The Dance, your job is to understand how much they know and how much they are going to be able to understand. How much experience do they have with the data, data analysis and the application of results to a business issue? You must also determine their definition of success for the project and what decisions are going to be made based on the results. Taken to its logical extreme, you can get them to pre-define their actions depending on the results.
If this KPI goes up by 25% we’ll do X
If this KPI goes down by 15% we’ll do Y
If this KPI does not change we’ll do Z
“Decisioneering” – coined by Chris Worland from Microsoft
Offering to improve business results will raise the profile of the analyst’s domain.
We can improve
by this percent
in this time frame
if we run this test
for which we will need these resources
There is a danger of diminishing returns when going after perfect data. It is not perfect and never will be. Analysts who spend too much time cleansing their data or continually cannot make a recommendation because there is not enough data should be retrained or replaced.
Analysis is about communication. Be sure to create reports that are accessible to the target audience. The New York Times got much more attention when they formatted their charts to look like New York Times financial page reports.
Define the problem. Given an hour to save the world, Einstein said he would spend 59 minutes defining the problem and one minute solving it.
Act like a recommendation engine. “You know, people who liked this report also liked this one.”
Never automate report distribution. It is an invitation to ignore the data. If they are not using the information to make business decision, they do not need to know the most visited page or the most popular exit page.
If you are going to highlight key learnings or keen insights, keep it to a minimum. Talk about two or three things at once – at most. The rest can wait for another conversation.
If you can match the behavior data with the attitudinal data, that’s good. If you can match the two of those to verbatim comments, that’s magic. People don’t respond as well to, “We have a 12% non-completion rate,” as they do to, “57 people told us they got stuck trying to change their profile settings and then we saw more than 750 were unsuccessful yesterday. We should do something.”
If you can figure out what report will help make your boss’s boss look good by making better decisions, you have gone from being valuable to being indispensible.
We started our doing analytics manufacturing. We are learning how to do analytics mining. Is it time to try our hand at analytics farming?
Analytics Manufacturing is all about the acquisition, refinement, assembly and maintenance of analytics as a continuous enterprise. Teams of specialists implement acquisition procedures and management strategies with a focus on production of large stocks of analytics with interchangeable labor. Yes, you end up with continuity of process which is easier to teach (and learn) and you gain some productivity. But this is a world of report monkeys – interchangeable skilled labor doing repetitive leg work.
Analytics Mining sees analysis as a valuable resource with insights to be extracted and refined. It requires solid tools to quickly extract value from a sea of data and rewards the individual analyst out sifting for gold nuggets. It’s easier to seize opportunity and act quickly. While manufacturing works with a “datawarehouse,” data mining sifts through that store of ore in search of the correlation that can enlighten the organization.
Analytics Farming looks at analytics as a continuing collaborative activity. It is performed by groups working together on changing individual and common goals and focuses on continuous cultivation and community. It’s a social activity where discovery and success are shared.
This communal approach puts the emphasis on the big picture and the meta understanding of the entire environment. Rather than raw material going in and finished goods coming out the other end, this is a delicate dance with the environment and with others in the organization.
More than just formulae and tabulations, analytics farming works in an ecosystem and a longer-term view. If the seeds (page tags) are properly planted, then the harvest (behavioral data set) can be reaped in a timely way, processed into usable ingredients and baked into a pleasing loaf of bread (valuable reports). If the bread is stored and distributed properly, it can be consumed as nourishment.
The analogy can drop all the way down to soil conditions (corporate culture), fertilizer (management sponsorship), rainfall (sufficient funding) and etc. The concept is that this is less a mechanistic or opportunistic endeavor, but a long-term it-takes-a-village enterprise.
To be successful at Analytics Farming, you have to foster an environment of collaboration, be willing to change with the weather, and accept that sometimes the crops are abundant and sometimes they fail.
Never hit people with a 16 ton sack of data. Make it palatable. Make it bite-sized. Don’t go too deep. Don’t baffle them with the reasons for your conclusions / hypotheses / recommendations.
Think of data like food. Too little leads to malnutrition. Too much leads to indigestion. Too bland is boring. Too rich leads to a mad-dash to the bathroom.
Always deliver context. A measurement by itself (I am 6 feet tall) isn’t even interesting. A measurement in context (I am 6 feet tall and trying to enjoy a tea party in a tree-house built for 8 year-olds) is interesting and leads to several trains of thought.
Host an event at your organization where you can teach analysis. Start non-threateningly with a brown bag lunch or an internal Web Analytics Wednesday and move up to an Analysis Boot Camp. Boot Camp should include a diversity of disciplines like user experience design, marketing strategy, business analysis, search, creative, and development.
Spend some time with a 4-year-old to truly grasp the meaning and the many uses of the question, “Why?”
Heed Descartes’ 4 principles:
- Accept only that input of which you are sure
- Divide a complex problem into as small parts as necessary
- Solve the simplest problems first
- Use induction and deduction to make as accurate and complete an inventory of the data as possible
Present yourself as an independent consultant who can look at things in a different way, from a different angle or in a different light. If you are an insider, make use of consultants – different consultants – from time to time to shake up your point of view.
A good analyst understands the business and knows whether the issue at hand is really about saving or making money this time.
Analysis is not a once-and-done endeavor. Times change, economies change, popular culture changes, competitors change. So you must embrace analysis as an on-going effort – a continuous process.
Understand the compensation packages of the people to whom you are delivering the analysis. To understand their motivation is to understand why they care about some facts and correlations not about others.
Market and sell the idea of analytics internally. Spread the enthusiasm as you would your favorite candy. Describe it’s goodness. Explain that it is worth the price. Offer a taste. Get them hooked. Send out small samples in personal emails. See who responds with the “That’s interesting!” look in their eyes. If you don’t see it there, back away slowly.
Market and sell your team internally. Put up profile pages. Show what they’ve done before coming to your organization and write stories about how they have helped others inside your organization. Explain what drives each of them and what types of problems they like to solve.
Don’t get tied in knots about not having enough data or having data that is clean enough. You never will and it never is. Perfect is the enemy of good enough. So while you’re waiting for better data:
Go after quick wins
Find ways to make others look good
Review the basics like search phrases and exit pages
Compare and contrast against other data sources like voice of customer
Good decisions are based on a logical process of considering the information at hand. A good decision does not mean the decision is right. Proper decision-making process that lacks insight, intuition and understanding is an algorithm. The human aspect is being able to consider things that are not necessarily part of the input set – like the weather.
The roles are often combined, but it’s valuable to remember that there are people who are better at some things than others.
Data, by themselves, are not actionable. Our tools are there to help us gather and manage the data but do not help identify the best way forward. They are not able to recognize and suggest a new, potentially better action to take. We must use our tools to aid in the understanding of the situation, but cannot expect those tools to help in decision making.
It is presumptuous to assume we can understand how the factors that actually matter in complex systems are related to each other and how changes in one impact another. The interconnectedness of elements and the law of unintentional consequences must always be kept in mind lest we imagine that we have infinite control.
Creating understanding is all about finding the right level of simplification – for your audience – to describe a system.
Are you reviewing and reporting on what your competitors are looking at on your website?
Deliver research and analysis that fits the current business process. They will not listen if the first thing you tell them is that they are doing everything wrong.
Centralize your data crunching and analysis at up front in order to get it organized. Next, decentralize as different business units understand the value and learn how. Finally, re-centralize in order to glean best practices and re-standardize across all departments. Repeat.
Look at the data, formulate a hypothesis, construct a model and then put it away for a day. When you come back to it, you will realize there are a few problems and will see four new possibilities.
When asking about business goals, do not settle for “raise revenue” or “cut costs”. Demand to know by how much and how soon. Now you will have a specific problem you can solve and the metrics will reveal themselves.
Get to know three key people:
- The person who controls the budget – they control all
- The person who decides how to achieve the goals – they need your help
- The person responsible for the outcomes – they will come to depend on you
Be willing to go on fishing expeditions. A total focus on outcomes is good, but exploration yields value as well.
How does your audience digest information? Some people like spreadsheets, some like charts, some like graphs and some like stories. Create your reports so they appeal to a variety of sense. The people inside your organization are your customers so segment them as to their learning style – then cater to it.
You don’t need to be precise, you only need to be compelling.
Visualization is a very persuasive tool which can more quickly illustrate relationships and changes over time. Periodically publish the most compelling images in a way that people can share. A picture is worth a thousand words and an animation is worth a thousand pictures.
Be a change agent. Read up on change management so you don’t risk too much but aren’t guilty of the sin of sitting on your hands.
A recommendation engine can be used internally as well. “People who liked this report, also liked that one.” Social rating is another form of recommendation. “207 people rated this report five stars.”
We aren’t decision makers, we are “decision enablers.” We analyze customer interactions with our websites, provide hypotheses about the world around us, deliver insights and suggest experiments. Our job is to help others make decisions by showing them the alternatives and suggestion which can be expected to have the best outcome.
Create a training and learning environment starting with interns and give people a clear, steady path to mastermind. Have a readily available chart of skills and required knowledge so people know what their career path looks like.
Do you know what tools you have available? Every four to six months, do a technology and data inventory. Other departments may have licensed a tool you haven’t seen before or are using a free tool. Some may be collecting a different type of data that might be enormously valuable when correlated with what you know you have.
Change all of your reports from the technical and tactical to the business and financial. Everybody in the company will understand a report about revenue & expense and profit & loss while only a few will follow a conversation about unique visitors vs. cookies.
When trying to effect change with data, use facts that are
Fun – something to get their attention
Interesting – something to capture their imagination
Useful – something to secure their enthusiasm
Go buy lunch for somebody in the finance department. Today. Just get to know them, understand their motivation and their perspective. Learn their language. Earn their trust.
Statistics are lumber – you get more out of them if you build something than if you hit people over the head with a 2×4.
“The most exciting phrase to hear in science, the one that heralds new discoveries, is not Eureka! (I found it!) but rather, “Hmmm…. that’s funny….”
- Isaac Asimov
Financial reports show impacts to transactions, but give little indication of where issues were occurring. Web analytic tools aggregate site usage stats but lack detailed customer experience data and information on customer intent. System logs and IT site monitors do not track details of customer experience issues. Code logging requires meticulous development efforts to identify and record every fork that could be exercised and it creates performance issues. Customer feedback mechanisms are not adequate and monitored or acted upon. Email is untargeted and drive spotty results. All of these tools are useless until we can integrate and correlate the data from them and then take some sort of action.
Until you have sufficient resources for the big datawarehouse in the sky, you can integrate data streams at the dashboard level and train people to do the pattern matching and critical thinking.
Begins with investigation
Leads to the uncovering of clues
Extracts meaning from clues
Assembles clues into a deduction
Enables the crafting of a story that narrates your investigation and final deductions
Generates a presentation, report, and additional docs that include recommendations
Your opinion though interesting is completely irrelevant. You are not the target audience.
Collect data for customers rather than about them. Use analysis to improve individual customer experience (targeting) and you will be rewarded. Use analysis sell more stuff and you will be frustrated.
First, look for anomalies.
Next, look for changes.
Then, look for patterns.
The more valuable segments tend to be small. Dividing visitors into male and female is helpful but the real gold is found in that fraction who share a half a dozen characteristics or behaviors.
When it comes to change management, do not preach to the choir – preach to the almost converted. Sing to those who are already humming.
Know what you don’t know.
This is what we know.
This is what we think.
This is the test we can run to find out.
Here’s what we learned from the test.
This is what we still don’t know.
Make a point of reaching out to the advertising people/department/agency at your organization and understand their process, their limitations and their motivations.
Measure the success of your measurement. Find a way to monitor how much people are using marketing analysis information to make business decisions. Prove that those who follow the results make more improvements than those that fly blind.
Prioritize the analysis you do depending on
Whether revenue is at risk
Who wants to know
How hard it is to answer the question
Whether it really requires an analyst or can be self-served How soon it’s needed
The purpose of the request
Occasionally take some time to ruminate about the data. Stir it with a stick and see if anything interesting floats to the surface. Look for patterns and correlations in the spirit of discovery. Sometimes the most interesting things come up when you’re just looking at the data rather than trying to solve a problem. But beware – this is dessert, not the main course. You must solve specific problems in order to earn the time to ponder.
Spend a day in the contact center. Take some calls about your website, talk to the service representatives and come to understand who your customers really are. Then you will be able to think of ways to integrate what data there is in the contact center with the data you already have.
Be ever vigilant against misoneism: a hatred or fear of change or innovation.
Most people use facts to confirm their beliefs. It’s human nature. They look for the data that supports their hypotheses. You are a trained analyst and have learned to guard against this practice but your internal customers have not. Those who come to you for answers may be looking for specific answers and will blame you for not doing your job if the actual data do not corroborate their desired results. You must teach them without insulting or alienating them.
“It is of the highest importance in the art of detection to be able to recognize, out of a number of facts, which are incidental and which vital. Otherwise your energy and attention must be dissipated instead of being concentrated.”
- Sherlock Holmes
It may not matter how often the customer has purchased before or it may be the most significant attribute of all. Finding quick ways to determine the relative weight of factors will help you cut through the noise faster.
When trying to understand a problem, ask for clarification. Ask the person on the other side of the table to elaborate. Ask for an example. Ask for the opposite of their situation. Ask them to show you what success looks like. Ask them to be more specific. Ask I they already have an opinion. Ask them how they will use the information they seek. What decisions will be made using the resulting analysis?
The number of insights is not proportional to the amount of data. There is no correlation.
Your job is part marketing, part IT, part finance, part market research, part customer service, and etc. That means you are a service organization and are only valuable you truly understand what each of those departments does for a living.
Etiology (alternatively aetiology, aitiology) is the study of causation, or origination. The word is derived from the Greek aitiologia, “giving a reason for” (aitia, “cause”; and -logia).
The word is most commonly used in medical and philosophical theories, where it is used to refer to the study of why things occur, or even the reasons behind the way that things act, and is used in philosophy, physics, psychology, government, medicine, theology and biology in reference to the causes of various phenomena. An etiological myth is a myth intended to explain a name or create a mythic history for a place or family.
“I don’t know – let’s run a test,” is often the most insightful thing you can say.
“Today’s scientists have substituted mathematics for experiments, and they wander off through equation after equation, and eventually build a structure which has no relation to reality.”
- Nikola Tesla
Analysis is not sequential at first. It starts with making unobvious connections and chasing hunches. It cognition, intuition and ingenuity. Then we can start on the path of collecting data, reviewing it and testing assumptions and hypotheses.
A good analyst always asks what problem they are trying to solve. And then asks the question behind the question. If they want to know the best way out of the building, it depends on why they want to leave. If they want a taxi to the airport, the front door is best. If they want to drive to the airport, the door to the parking lot or garage is best. If they want to take out the garbage, the back door is best. If the building is on fire, the solution is very, very different.
Think of data as evidence. There is inconsequential evidence, distracting evidence, circumstantial evidence, etc. There is sometimes the need to know something beyond a shadow of a doubt and sometimes one only requires a preponderance of evidence. Evidence should be respected – one must always be careful not to contaminate the evidence.
“I do not paint things. I paint only the differences between things.”
- Henri Matisse
Just like web visitors, people who consume your reports and view your dashboards cannot be bothered to scroll.
No matter how many people you have and how talented they are, periodically bring in an outside consultant. They come with a different perspective, different experience and are willing to ask those critical, random questions that scare everybody on your team.
Business acumen comes from having sufficient knowledge about organizational goals and what everybody else in the company does in order to use your superpowers to help them achieve their goals. Simply providing reports to people only gives them the ability to ask questions in just the right way to cover their own backsides.
Hire people who like puzzles, detective novels, science fiction, crosswords and whose eyes light up when they learn something new.
When it comes to segmenting customers by behavior:
- Some do and some don’t.
- The differences aren’t that great.
- It’s more complicated than that.
- Bernard Berelson
Human Behavior: An Inventory of Scientific Findings (1964)
What follows are a few things I’ve picked up since the eMetrics Analysis Symposium. I hope you find them useful.
A website without metrics is just a hobby.
- Peter Andersen, Danske Bank
We are using analytics as a treasure map to fund our search for truth.
- Matt Cutler & Brett Crosby, X Change 2009
We are not accountants, we are statisticians. Statistics means never having to say you’re certain.
- Bob Page, X Change 2009
The truth changes depending on where the tag is on the page
- John Pastana, X Change 2009