I continue to be vexed by internet gremlins. I couldn’t post this comment as a comment so I’m taking Eric’s advice and am posting it as a new post. Sorry, folks.

Anyway, this is a response to Debbie Pascoe’s 16 Aug 08 11:22am comment.

Yes, owners not tagging all the pages desired, intended or necessary. We share that evil, I guess.

I didn’t know so many possible errors could creep in as far as javascript goes. It makes me happy that one of our original design decisions to was make things as simple as possible on the client’s side and that all the heavy lifting would be done on our side. Our side we can control pretty well. The client? Not so much so.

Thank you for describing what is meant by calibration (”automatically scanning the site and uncovering these conditions so they can be corrected”). This is something our technology does during the running of reports (uncovering invalidating conditions).

Calibration should not be a core competency of WA vendors? Yet WA vendors should partner with people who have that as a core competency. Hmm…

So if they should partner with such individuals, they must in some part be assuming responsibility for proper calibration, correct?

I read your Response to Michael Wexler’s Post re:What Web Analytics is Missing and Michael Wexler’s What Web Analytics is Missing… (with any luck I left a comment there). I asked some NextStagers about this. The discussion was very interesting. The core issue is something we’ve seen repeatedly in labs; people making first order estimates based on inadequate data because they failed to figure out the basic parameters of their experimental systems.

After a lot of conversation it pretty much came down to “Yes, calibration should be something offered by vendors” and there was a caveat that plays to your partnering theory, that such a service should be included in the fee structure.

The fact that this topic has risen to the “conversation topic” level is an indication that it happens enough to be recognized in the web analytics community. Of course, one of the joys of our society is that we think contracts assign and absolve responsibilities (except probably with very large clients).

Lucky us, huh? Sometime when we’re together, remind me to tell you what one of our early employees, a fellow from Australia, thinks about contracts in the US.

The amusing piece of all this is (to me) that (I thought) web analytics was all about accountability.

Glad you enjoyed the thumb comment. I only wish it weren’t true so often.

Howdy All,
this post is a response to Visitor Engagement: Time for a reality check? posted by Matt Belkin, 14 July 08 on Industry Insights The Omniture Blog. A friend sent me the link and asked for my thoughts. I read the post, formed my thoughts, asked some folks to review what I wrote, did a little editing and went to post it today. I clicked to post my comment and my screen went blank (quite common for me) hence I’m posting my comment here and on Responding to “Visitor Engagement: Time for a reality check?”. Susan Bratton calls me a Righteous Luddite. Not sure about the righteous part, definitely a Luddite.

Anyway, here’s what I posted as a comment to “Visitor Engagement: Time for a reality check?”

Hello,

If I read the paragraphs under the The Basic Premise of Measurement heading correctly the logical outcome is that the only measurements being made (or provide value) are those that are subject to intervention (”improve”).

I have to admit that confuses me. I agree that lots of things are measured with the goal of acting upon the results of the measurement. I also recognize that lots of measurements are made simply to know what they are and with no realistic hopes of being able to modifying what’s being measured. I did look in several texts (started with Horst De la Croix’s Psychological Measurement and Prediction, went to Sabins’ Remote Sensing: Principles and Interpretation (I’ve often thought this discipline most closely demonstrates concepts and principles that would be valuable to people doing web analytics. I mean, consider Sabins’ opening statement to the 2nd edition, “Remote Sensing is broadly defined as collecting and interpreting information about a target without being in physical contact with the object.” Admittedly website visitors aren’t the subject of the book and the theoretical constructs often apply, I think), skipped over to Fukunaga’s Introduction to Statistical Pattern Recognition with lots of side trips into other fields) to find something similar to “The basic premise of measurement is that you want to measure something so you can improve it, if necessary.”

The best I could come up with was that the basic premise of measurement is to determine if something is measurable. I think the reframe is valid and leads to some necessary concerns.

Example: Some factor, A, is recognized and known to exist within a system (a website). A measurement is taken. Modifications are made to the system and another measurement is taken. But factor A hasn’t changed. Perhaps something else has changed and not factor A, the recognized and known factor of concern.

So we recognize that whatever is being modified based on the above measurement has no effect on factor A hence whatever is being measured is not factor A nor does it contribute substantially to factor A.

Yet still we want to understand factor A and be able to effect it at will. Other measurements are made and only of things we know we can improve. Nothing in our existing measurement framework seems to affect factor A.

Shall we continue? Shall we pursue other metrics? Shall we investigate until we can produce a “this = that” metric that has statistical validity?

“The basic premise of measurement is that you want to measure something so you can improve it, if necessary.” Thus the answer is no, we neither pursue nor investigate.

Nothing in our existing repertoire successfully produces a “measurement so you can change it” tool for factor A therefore we can conclude any of a) factor A doesn’t exist (even though it is recognized and known), b) it exists and is unmeasurable, c) the existing toolset does not include a tool with the ability to take a measurement of factor A such that changes can be made to the system that affect factor A, …

However, the fact that what is measurable (and actionable based on those measurements) is changing is quite true. The horizon of metrics meeting the initial conditions of measurable and actionable is, like the observable universe’s, growing every moment.

And I freely note that I’m a researcher, not a web analyst. I did ask one of our researchers, someone with a long history of lab work, and they offered that the basic premise of measurement is to determine the fundamental characteristics of what is measured.

“It’s pretty simple — collect data, analyze, improve. I love this because of its simplicity and objectivity.” But if the only data being collected and analyzed is based on factors we know we can change then there is no objectivity in the measurement or method involved. The only factors being analyzed are those recognized as changeable in the existing paradigm (the toolset).

Let me offer an example that demonstrates moving from measurable but unactionable to measurable and actionable: We’ve been intentionally collecting meteorological data for about 250 years. There was nothing in that toolset that could affect climate until recently. We also developed the ability to collect meteorological data going back far into prehistory (I can provide references to paleoclimatology if necessary. A quick search of Science provided some 1012 references going back to 1900).

Thus things were measurable. Without being able to measure them we could never have learned a) that they did indeed exist, b) that they were affecting us or c) how to affect them. Thus does our observable (web analytics) universe grow from measurable but unactionable to measurable and actionable.

“…you introduce a level of abstraction on the data. You “dumb it down” introducing bias and subjectivity.” I agree with this to a point and offer Communicating Science to Business and Vice Versa for your reading pleasure. I also recognize that all web analytics solutions offer training and certification on their toolsets.

Are these trainings and certifications offered to insure that all abstraction, bias and subjectivity are removed when a consultant or in-house analyst uses a given toolset? I want to take part in a web analytics tool class where the word “interpret” isn’t used.

“So what happens when you start combining metrics into uber-formulas …? That model breaks, because you introduce a level of abstraction on the data.” This also confuses me. Given a formula that has several elements and each element is known to be valid and measurable then the sum of the elements is also valid and measurable. This is basic mathematical logic (distributivity); if x is true and y is true and conservation of units holds then (x + y) is true. This principle (not by name) is mentioned in this post itself when the subject is RFM.

“First, what kind of return frequency is “often” - two visits? Four? Six? That’s subjective. What is “important” content? The home page? An article? A support document? Subjective again. And what is a “long” time on site — 5 minutes, 10 minutes? Perhaps “long” means any visit that exceeds the average for the site that week?” This logic reminds me of one of my favorite jokes:

---

A man wants to know what 2 + 2 is so he goes to a mathematician and asks, “Professor, what is 2+2?” to which the mathematician replies, “How may decimal points do you want it out to?” He then finds a psychiatrist and asks, “Doctor, what is 2+2″ to which the psychiatrist replies, “Interesting. How did your mother treat you when you were two?” He then goes to an economist and asks, “Sir, what is 2+2?” and the economist answers, “What do you want it to be, my boy?”

---

I do agree that the types of values you reference need to be defined, qualified and quantified (and my experience is that they usually are by the business model of the organization requesting the metrics to be applied) before being used in any measurement and resultant decision process. This is something I believe Nathan Janitz states in his comment(s). I also believe that to suggest subjectivity is not part of the business world — even when it comes to something as definite as well-defined measurements — is naive. To that point, suggesting that “often”, “important” and “long” implies subjectivity without recognizing that the conversation drawn from was not meant to have any rigor is amusing, hence my offering of the above joke. This goes back to the reference to RFM. If the underlying (or “principle”) metrics are valid then the uber-metric is valid. And I also recognize that my paradigm might not be appropriate here. I do think the core concept of measurement (of anything) is scientific in nature. Ms. Debbie Pascoe writes in her Continuing the Discussion with Joseph Carrabis “Anyone who has ever used a tire gauge or a tape measure has employed a scientific calibration method.” Hence I offer Paul Davies’ “A scientific claim is taken seriously only if it can be tested by others in a disinterested (not uninterested) way.”The true test of any measurement is that it can be duplicated when attempted under similar situations with similarly calibrated tools. That concept, I believe, is where true simplicity and objectivity will demonstrate themselves first. After that is achieved, companies that make measurement tools will find a way to make the measurements personally relevant to consumers (see Framing Science). This, I think, hearkens back to Ms. Pascoe’s comments and thoughts.Jim Novo offers “Simple to explain to just about anyone…” and I worry if our decisions on usefulness are going to be based on simplicity of explanation. If so, then everyone who doesn’t understand solid-state physics, magneto-interference pattern imaging, magneto-optical coding, lasers, satellite telemetry, … oh, heck, let’s just go to basic electricity… throw away your digital cameras, computers, pdas, smartphones, stop watching TV, move out of your homes completely, never again use cars, buses, boats, trains or planes, …

But (!!!) if the decisions on usefulness are to be based on simplicity of use? That’s a completely different story. My experience is that most people don’t care (or even care to know) how complex something is “under the covers”, they just want to know that “when I do this, that happens” and even more accurately “when I do this, I get what I want” (this is the history of technology concept I often share (see The Long Tail, Part 1, Rocks, Hammers, Competition and How People Get Left Behind or VerizonWireless’ 20 year plan)).

Bravo to some and not all of what Steve Jackson adds. In the end, any metric that provides value will survive, even if that value is simply that something is being measured.

I also appreciate Nathan Janitz statements. His statement “…you also can’t catch a mouse without looking at the right information….all of the right information” is, I think, exactly to my point.

Allow me to quote Joe Tragert, Director, Market Development, EBSCC Publishing, ‘The question isn’t ‘Which mouse trap works better?’, it’s ‘Did we kill the mouse?’” You can have all the metrics in the world available to you. And if your goal is still unachieved? Or if none of the available metrics address your goal? Then it is time to either give up your goal or create new and valid methods of measurement that help you achieve that goal.

Proving that Serendipity is doing it’s job, I’ve had in my mind that it’s time to return to these thoughts and several people contacted me to find out if I was going to return to this blog.

Okay. Into the deep end first.

My time away has been due to busyness. Perhaps some readers have heard, NextStage Received its first patent on its Evolution Technology. For years we’ve been intentionally below the radar, now we seem to be becoming a recognizable object rapidly approaching from the far horizon. Now that we’ve left nap-of-the-earth flying I’m able to discuss things more openly, me thinks, hence some of my responses now and in the future.

Are the visitors happy?

One of the things I did while I was away was talk with a few people (about 100 so far) about what I’ll call The Purpose of Web Analytics. I did this research because of something I wrote in this thread above, “…all these analytics are worthless unless they create happy, satisfied visitors, yes?”

I’ve talked with upper management in education, politics, at national telecoms, financial institutions, transportation, recreation, … a pretty diverse group. Most of them were involved in marketing products or services or some other form of gaining marketshare. None of them were web analysts or involved in web analytics except that they received reports and were expected to act upon them. None of them were particularly happy about being made accountable to a system that (they believed) wasn’t measuring … and here’s where the challenges really made themselves known.

What was being measured? Lots of money was being spent and lots of people were being told that the measurements mattered and as one fellow explained, for the amount of money they were spending they expected some consistency.

“What do you mean by consistency?” I asked.

He pretty much didn’t know. He and those with him said lots of things and it could be distilled to a general dissatisfaction that there wasn’t a single model that they could consistently use and derive actionable meaning from. The dissatisfaction grew geometrically when the discussion got into executives making decisions based on sales presentations rather than a given product’s specific informational abilities.

At one point I leaned towards a speaker and quietly said, “Remember, Joseph friend,” and everybody laughed because the tension in the room was broken.

I reference these anecdotes because one of my original hopes for this platform was an increase in understanding and acceptance of some mutual goals regardless of discipline or tool platform.

In the end, doesn’t it all come down to “…all these analytics are worthless unless they create happy, satisfied visitors…?”

If I can’t act on it, it doesn’t exist

The next item I wish to thread into this discussion comes from an online conversation I had with Critical Mass’s Christopher Berry about why web analytics seems to be a harder sell in Canada than in the US. You can follow my side of the conversation in Canadian Based Business Differences — Responding to June Li, Christopher Berry and Jacques Warren, Responding to Christopher Berry’s Vexing Problem, Part 3 post, The Language of Web Analytics - The Hard(er) Sell in Canada, Responding to Christopher Berry’s “A Vexing Problem, Part 4″ Post, Part 1, Responding to Christopher Berry’s “A Vexing Problem, Part 4″ Post, Part 2 and Communicating Science to Business and Vice Versa and links are provided to Christopher Berry’s side on the conversation in those posts. I’ll invite people to pay particular attention to Communicating Science to Business and Vice Versa because (and as Mr. Berry noted) the summation is what counts, “Business is different. Business (me thinks) tends to be more ‘Tell me how to use this’ hence most business proposals and reports start with Christopher Berry’s nuggets then go into explanations.”

My research is convincing me that (what I recognize as traditional) web analytics is going to be losing its authoritative power in the coming years. I think web analytics (and yes, this does go back to my original hopes for this blog) will evolve (just as anything will if it is going to survive in a given changing environment). What will it do and look like? I have some ideas, of course. Just ideas at present, though. More things to research before putting down on paper (or in a blog) at present.

This does tie into my comment re Avinash Kaushik’s “…we shouldn’t use ill defined engagement metrics as a proxy for something solid like a sale.” I’ve been an oft-times unwilling father-confessor to businesses frustrated by ill-defined metrics of any kind and wanting something that is justifiable a) financially, b) scientifically, c) arithmetically (forget mathematically) and d) produces some kind of “do A, get B”, “this-equals-that” link between action and outcome.

The comment I love about this is “If I can’t act on it then it doesn’t exist”, ie, it’s noise, a distraction at best and something best ignored. This was a wonderful statement used in a business practices discussion.

I’d really enjoy being involved in a web understandability/measurement/future usability discussion that has as its theme “If I can’t act on it then it doesn’t exist.”

“To measure and analyze on and offline behavior and then try to predict who to market to by figuring out what they think is not doable with one tool or one metric.”

I responded earlier to this comment. People who attended either the Toronto ‘08 or SF ‘08 eMetrics conferences are probably well aware by now that NextStage has patented a technology that can determine how someone is thinking through any programmable device. I won’t go deeper into the topic here except to offer a comment I posted on Jim Novo’s blog about the {C,B/e,M} matrix and its use in marketing and analytics.

Picking up where I left off with Jim Novo’s comments in this thread…

I finally had an opportunity to read Jim Novo’s Measuring Engagement and its related Framework for Engagement posts. I truly enjoy Jim’s writing style and the points he makes.

I especially enjoy and appreciate his referencing Relationship Marketing because it places people center stage. Understand people and you can both understand and predict what they’ll do. Watch only what people have done and you can only understand their actions in a specific historical context, you can only predict what they’ll do when the confluence of events that led to their original actions repeats itself. Exactly (and don’t hold your breath). Relationship marketing works at the question “…all these analytics are worthless unless they create happy, satisfied visitors, yes?”

Jim writes “The challenge with this model - and probably why it isn’t more widely known - has been the data, it’s a very analysis-intensive model…”. Yes. Agreed. If Jim (or others familiar with these concepts) is reading (or perhaps at the next conference we meet at), I think this is where being able to substitute cognitive heuristic models makes sense (see Liberation and Heuristics or Responding to Christopher Berry’s “A Vexing Problem, Part 4″ Post, Part 1. I’ve also written elsewhere that I often wonder why more businesses don’t make use of cognitive heuristic models).

For example, I’ve recently been applying heuristic models to helping adult second language learners increase their language acquisition abilities. That’s a traditionally very tough nut to crack and (so far, anyway) I’ve been able to isolate neural activity that tends to make adult language acquisition challenging. Example 2, using heuristic models in the above grew out of learning which heuristic models are used (non-consciously, of course) by which personality types in their decision making processes. This non-conscious heuristic model selection process is being integrated into NextStage’s Rich Personae. These and some other areas of my studies are intensely data-rich models that can be reasonably simplified via cognitive heuristics.

I also strongly like your concept of dis-engagement, although I tend to use a methodology that incorporates “satisfaction” into the scaling system (see Meet Online Engagement’s Little Friend, Satisfaction. I shared that the complete form of this during a discussion at the SF ‘08 eMetrics. It looks something like the figure on the right.

Some definitions to help in understanding; the x-axis is Engagement and is a measure of the amount of pleasure or pain an activity is giving you. If something is giving you either pleasure or pain to any degree your attention is focused on it, hence you are engaged by it according to the definitions documented in Attention, Engagement and Trust: The Internet Trinity and Websites. The y-axis is Satisfaction and is a measure of acceptance and rejection of some internal state and/or external event.

I believe what you are referencing as “dis-engagement” is what we recognize as the slide from high acceptance to “0″ acceptance. Note that this is not rejection (as rejection is an active negation of acceptance) it is a lack of acceptance. I appreciate that the difference might be subtle and I believe that difference is significant. Rejection — the active negation of acceptance — can be thought of as someone pushing something away. Zero-acceptance is the point where one can “take it or leave it” and the internal state and/or external event does not have any value assigned to it, hence doesn’t register strongly in the mind/brain.

Mapping this figure to real world experience, you always want visitors/consumer/etc to be in the first quadrant (where the green curve is). People are both positively engaged (they like what’s going on) and positively satisfied (they accept it gladly). Depending on what you’re selling you may or may not want people in those other quadrants. The second quadrant (bottom yellow curve) indicates someone focusing on painful experiences or information, the fourth quadrant (top yellow curve) indicates someone who finds pleasure in painful experiences or information. The third quadrant (red curve) is where visitors/consumers/etc often end up and marketers/businesses don’t want them to be — the former are actively psychologically and physically moving themselves away from a business/product/service.

I’ll offer that the above is also a reasonable representation of your:
1. Define / Measure Engagement – any way you want to, as appropriate for your business; whatever activity or combinations of activity you feel appropriate
2. Measure dis-Engagement – the absence of Engagement, as in the visitor / customer stopped doing whatever it is you define as Engagement for your business model

I think where the image above (and the math behind it) adds real value is with your “3. Take some kind of Marketing or Service action to slow or reverse the dis-Engagement with dis-Engaging folks” because it provides enough information to know how, exactly, visitors/etc are “right now” interacting with your marketing information.

I also agree whole-heartedly with your statements about predicting “dis-engagement”, etc. I would love to see the data you used in your example and apply it to the above. I’m willing to bet that satisfaction/acceptance was the real driver (and I won’t get into the depths of group satisfaction/acceptance states here (really, Joseph? You’re going to leave something out? Whatever for?)). I did get a kick out of your graph of email response rates falling over time. It was very similar to the results we found in our research on how to design an effective email newsletter. Bravo! I always love it when our findings match others’. Gives me hope we’re doing something right.

<ASIDE>
For what it’s worth, much of the rest of what you’ve written in your post is so close to what we learned in our email newsletter research that the overlap is astounding. Not surprising, I guess, as you’re listing an email-based experiment. It would be interesting to learn what else the rules we discovered pertain to. Let me know if would like to explore this.
</ASIDE>

You also list an implication about sending different messages to different segments. Yes, agreed. I believe the above allows for much more targeted and action-driven messaging (based on much of what I’ve shared above).

Perhaps, in the end, we’ve derived nothing more than a simplified mathematical model (complete with suggestions for better outcomes) of Relationship Marketing?

Whoosh!

Took me two days to put the above together folks. Sorry for the delay. More to follow. Soon.

Promise.

Based on my experience, the majority of web analytics (WA) tools are currently managed by a single, in-house person.  More than likely, the WA tool is underutilized and the WA team is too small and undertrained.  Heck, web analytics is hard. ;)  I predict that within three years, media agencies will build out ‘web insights’ specialties and capitalize on the web analytics opportunity.  Why will they be successful?  (Disclosure: I work for an Omnicom Agency)

Current Lack of In-House Ownership

Where does WA expertise & ownership reside in your organization?  Marketing?  IT?  Somewhere in the middle?  Both?  There is no consistent answer.  And I doubt everyone in your organization that could find value in WA data uses it (or even knows it exists).  Rarely is WA tied into major company marketing priorities, but most analysts agree it should be.  The proverbial WA ball has fallen (or will fall) through the cracks somewhere between Marketing, IT, Web Dev firms, and interactive agencies.  My argument: the group spending the most money to drive web traffic should “own” the full user experience and thus, web analytics.  Typically, this group is an outside agency focused on display, email, paid search & SEO.  Website performance, usability, and page optimization has a large impact on these online investments and are factors directly tied to performance/ROI and overall marketing strategy.

Current Lack of Talent & Economies of Scale

There are not enough talented analysts to meet the current demand, much less the future needs of our data-driven, B.I.-focused industry.  “Experience”, although important, is not the same as talent.  Similar to SEO, scarcity of talent leaves a large gap between those who dabble and those who specialize/excel.  Independent specialists/consultants will realize that nearly the same WA tactics (data insights and actions) have major gains for clients and economies of scale are found.  To get access to the top tier of clients (and dollars), these specialists will either join or become acquired by large agencies to add additional value to the current agency optimization techniques.

WA Insights Improve Agency Work

Tastes Great or Less Filling?  …Put it on the website and see what visitors respond to more.  If a company invests in a regional TV/Radio media buy, what results were seen online from the area – and was it ROI positive to justify expansion into new markets?  For the Fortune 500, agencies are firmly entrenched in building the websites and driving the traffic to increase the company’s revenue.  With WA being the ultimate tool of understanding the online experience and improving it, shouldn’t agencies put it to work to support their vision?  I can’t think of a more appropriate group to gain insights and take action from the data that results in a major impact to the entire business.  When you can make millions of dollars in online spend convert 1% better by improving the cart process, you just made the client a heck of a lot of money and the agency more valuable to the client by increasing the media ROI.  Further support of the growing importance of the media agency was seen in AdAge’s “Why You Should Be in the Media-Agency Business.”  The article cited a recent Booz Allen Hamilton study that asked marketers which organizations would become more important to them by 2010.  Media companies, media planners and communications planners topped the list, with 52% of respondents believing they would be more integral.

Media strategists will seize the WA opportunity.  Clients are demanding agencies to be more accountable for online performance as well as more data-backed “proof” for business strategies and tactics.  The recent trend of the digitalization of offline media will only increase the need for talented analysts to interpret the data for the media agencies.  Only time will tell.  In the meantime, please share your thoughts.

Jeff Campbell
VP, Product Development
Resolution Media