Recently Justin Kirby of Viralmeister posted on his blog, and left a comment on my blog post, regarding concerns he had about the methodology of my research report "To Tell Or Not To Tell?" So that other interested readers could follow along I wanted to "promote" his comment to its own blog post and respond. I apologize in advance for this being a long post but I think it's important to address each concern in detail (and besides, it's an occupational hazard for me to provide verbose explanations -- ask my students, they'll tell you).
Justin was responding to my post on a recent study by Alain Samson at the London School of Economics (I pull excerpts for the sake of space; see Justin's full post):
... To his credit Dr Carl does point out that not everyone is cheer leading the Net Promoter Score and makes some valid points about the need to see the methods and results used in the analysis in order to assess the merits of this latest research from the London School of Economics. ... Anyway, I was mildly amused about Dr Carl’s stance on methodology because I felt the same way about his research on Bzzagent’s model (To Tell or Not to Tell), which by his own admission was based on Bzzagent’s internal analysis of the 270,000 word-of-mouth reports from their own agents. I’d love to see how Dr Carl adjusted for bias of not only Bzzagent’s internal analysis of the reports, which is far from objective, but also the bias of the reports from a ‘trained’ group who are both directed and incentivised to spread the word.I'd like to take the opportunity to respond to Justin's comments to clarify his valid points from apparent misunderstandings of my research. For a quick overview of the main findings of the To Tell Or Not To Tell? report please read here (the full report can be downloaded as well).I just wonder where the control group was because I’m dubious about solely using a panel which has been designed to leverage the Hawthorn[e] Effect rather than a more conventional research panel that tries to adjust for this kind of bias. Maybe Dr Carl could put me right on this because I can’t help thinking that facts were fitted to theories about disclosure which fly in the face of Attribution Theory.
First, the To Tell Or Not To Tell? (TTONTT) report was not based on "BzzAgent's internal analysis of the 270,000 word-of-mouth reports from their own agents." This was mistakenly reported in a ClickZ article entitled "BzzAgent to Agents: Spill the Beans, Or Else." I wrote a blog post to clarify the mistake in that article, which Justin linked to, so I would encourage Justin and other interested readers to re-read that post. As I wrote in that post, BzzAgent prepared their own white paper citing internal reports of their agents; none of the findings from TTONTT relied on those reports so there was no bias to adjust for regarding a separate analysis conducted by BzzAgent.
Second, Justin was concerned about the "bias of the reports from a ‘trained’ group who are both directed and incentivised to spread the word." Here again, this is a misunderstanding of the TTONTT methodology. I employed a dyadic methodology that relied on surveys completed by BzzAgents (not the internal reporting done by BzzAgents as part of a campaign, but surveys completed as part of this specific TTONTT research project) and their conversational partners (the people they talked with about the brand, product, or service). To account for any potential bias in the BzzAgent's responses to the TTONTT survey we validated their responses with their conversational partners' responses. So, for example, if a BzzAgent said they disclosed but the Conversational Partner said they didn't disclose, this was noted as a discrepancy, and the discrepancy results were fully reported and accounted for in the analysis on pages 10-11 of the report.
Third, Justin wondered where the control group was because he is:
dubious about solely using a panel which has been designed to leverage the Hawthorn[e] Effect rather than a more conventional research panel that tries to adjust for this kind of bias. Maybe Dr Carl could put me right on this because I can’t help thinking that facts were fitted to theories about disclosure which fly in the face of Attribution Theory.Justin's point about a control group is valid and I appreciate the opportunity to respond. The TTONTT report was part of a larger project where we were trying to understand multiple perspectives on the same interaction -- the BzzAgent's and the Conversational Partners' (many studies rely on just one person's perspective) -- and also to determine how Conversational Partners were affected by talking with a participant in a word-of-mouth marketing program. And then, in addition, the study was to look at what role disclosure of the agent's affiliation with a WOM marketing company might be.
The study did not employ a control group where we gave instructions to some agents to disclose their identity, other agents to not disclose their identity, and then another group where no instructions regarding disclosure were given (the third group here could be used as a control group). The reason for not doing this was because it would have violated BzzAgent's policy surrounding disclosure, which required agents to disclose their identity (see page 8 of report). Instead, what we did was to conduct the analysis by comparing two groups after we collected the data: 1) Conversational Partners who knew they were talking with someone affiliated with a word-of-mouth marketing company and 2) Conversational Partners who did not know the agent's affiliation. NOTE: I relied on the Conversational Partners' responses (that is, non-Agents) for most of the analyses, except when I conducted the discrepancy analyses where I matched the BzzAgent's survey response to the Conversational Partners' survey responses.
By doing this post-hoc analysis, rather than using a field-based quasi-experimental design, or a laboratory-based experiment, this study has limitations, as all studies do (and there are other limitations to the study as well, all discussed in the report on pages 20-21). For example, we found that there was no difference between the outcome variables between the two groups (no difference in a conversational partner's likelihood to inquire further, to use the product/service, to buy the product/service, or to tell others about the product/service). But we did find that people who knew of the agent's affiliation told more people about the product/service. Because we didn't use an experimental design we can't conclude that disclosing agent affiliation led to higher pass-along or relay rates (more people being told). We can only say that conversational partners reported higher pass-along rates in conversations where they knew they were talking with someone affiliated with a WOM marketing program. However, while noting this limitation there are a number of important results that are, as Justin rightly points out, at odds with what we would expect from attribution theory (see my blog post responding to some of these counter-intuitive results; for readers unfamiliar with attribution theory as it relates to WOM, see Greg Nyilasi's chapter in the Connected Marketing book that Justin edited with Paul Marsden).
I was surprised myself by a number of results from this study -- this is the great thing about conducting original research -- so it would be inaccurate to say "that facts were fitted to theories about disclosure which fly in the face of Attribution Theory." But when you dig deeper into the analysis you find something that's pretty interesting and it's that attribution theory may still apply, as long as you take into account the relationship between the BzzAgent and the Conversational Partner. Here's what I mean...
If you're talking with a stranger or acquaintance -- people you don't know particularly well, or at all -- and the only thing you know about them is that they're part of a particular kind of WOM marketing campaign, then you might be more likely to question the person's credibility to give an unbiased opinion or an opinion that may not be in your best interest (in fact, some of the data I had about interactions with strangers actually trended in this direction; however since most BzzAgents speak with friends and family members (see page 6), rather than going up to strangers, we didn't have enough strangers to make valid statistical comparisons). However, if you know a person in a range of different contexts and have talked with them before, and know from those interactions that they generally have your best interests at heart, you're much less likely to question their sincerity when they share their opinion about the brand, product, or service. I think that because there was a high number of "stong-tie" relationships between the BzzAgents and Conversational Partners, this explains a good bit about why the results turned out the way they did (both BzzAgent's internal research and my own research partnering with them shows that the majority of the conversations are with already-known others).
Finally, Justin also expressed concern about a using a business model that's designed to leverage the Hawthorne Effect (meaning that people's behaviors will be affected by the act of giving people attention and making people feel more involved, which is what many WOM markting programs seek to do in order to stimulate WOM; interested readers should see Paul Marsden's chapter on product seeding programs in Connected Marketing). Here, again, I would reiterate that I surveyed Conversational Partners, in addition to the BzzAgents, who were not affiliated with the WOM marketing company. I would also offer that the study should be repeated with a wide range of different models and techniques of WOM marketing programs.
I hope that there are still a few readers who have made it to this point of the blog post! :-) I apologize for the length of this, but I appreciate the opportunity to clarify the study and I invite others to challenge the results and engage with the study so that we can achieve a better understanding of the role of disclosure in WOM marketing programs.
Justin, does this address all of your concerns?
Download the full version of the To Tell Or Not To Tell? report, as well as other papers I've written, for free at my download page
UPDATE (01/22/2007): Justin Kirby has posted a response to my response.
-->
Tags: WOM word of mouth Word-of-Mouth Marketing buzz marketing viral marketing marketing communication
|