This upcoming Friday (3/5) I’ve been invited to speak to a Marketing Research class at Skidmore College. The purpose of my presentation will be to show the students all the different ways research is used in Media Planning and Buying and the associated problems/challenges with that research. Below I’ve drafted an outline of the presentation and for this blog entry I’d like to ask for feedback (questions, comments, additions, subtractions) to help make this the best presentation possible.

Marketing Research: Presentation at Skidmore College Friday 3/5/2010

  • 1. The advertising equation (follow the money)
    • The main purpose of all advertising is to make incremental money for the Advertiser (ROI).
    • Typically the Advertiser gives a sum of money to the Agency, the Agency takes a fee and buys media from Publishers, and the Publishers help the advertising reach Consumers.
      • The goal is to get the Consumers (reached by the advertising message) to give more money back to the Advertiser, than was spent by the Advertiser to reach them with the message (the goal is positive ROI).
  • 2. John Wanamaker (quoted): “Half the money I spend on advertising is wasted; the trouble is I don’t know which half”
    • The purpose of all Media research is to make sure advertising dollars are not wasted. In order to avoid wasting advertising dollars, research helps us select media vehicles that reach the highest portion of our target audience and no one else (it costs money to reach each person – money spent to reach people outside your target is wasted).
  • 3. The three stages of Media Research and their associated problems/challenges
    • 1) Pre-Campaign Research
      • This is the research done by agencies to make sure they buy the media that best reaches their audience with no waste (reaching people outside their audience).
      • This research relies on two sources 1) National consumer surveys (Simmons, MRI) and 2) Ratings agencies who calculate audience for TV shows, magazines, websites etc… (Nielsen, comScore).
        • Problem 1: The Simmons national survey is a 185 page workbook with over 1000 questions – would you fill it out? (response bias may be a factor)
        • Problem 2: Nielsen bases TV ratings off of “C3 ratings” (commercial ratings, plus three days of DVR playback) – is this really an accurate way to measure audience (everyone knows people watching shows on DVR don’t watch the commercials)? Also – there are only “people meters” in the top 56 US markets – all other markets are measured by paper journals – which introduces more bias into the measurement equation.
    • 2) During Campaign Research
      • This is the research done by advertisers and agencies in order to make minor course corrections (optimizations) to maximize media impact.
        • Problem: For branding campaigns optimizations are based on KPIs (Key Performance Indicators) which are frequently a leap of faith away from true ROI (e.g. shifting spend toward Banner Ads with the highest CTR (Click Through Rate) may not actually be helping your campaign – it could just be moving money toward people who tend to click on ads – research suggests that only 8% of the entire online population generate 80% of total ad clicks).
    • 3) Post Campaign Research
      • This is the research done by either 1) Test/Control survey methodology to see if people remember seeing your ad or 2) Examining campaign metrics to make an inference about what was successful from an ROI perspective.
        • Problem 1: Post campaign surveys are often rushed and based on small sample sizes – also results usually trend toward positive as all parties involved benefit from positive results (including the survey company who will likely be rehired to measure the next campaign).
        • Problem 2: No matter how you manipulate campaign metrics (KPIs) they’re still a leap of faith from actual ROI.
  • 4. Class Discussion Topic: How do you know your advertising is working for you? (is it possible?)

Thank you for reading through this outline – any and all comments and suggestions are welcome!

The Role of Research in Media Planning and Buying
  • It’s a nice set of the measurement problems involved in marketing research.

    One question: to what extent the company evaluate ROI by changes in sales? e.g. they could create a model of what sales would be without advertising, and then see what change there is – and after accounting for extra-model shocks, the rest of the change should be from advertising. Naturally, this method is rife with problems, but I don’t think it’s something you mentioned.

    Re: Post-campaign surveys are always based on small sample sizes – the small sample size shouldn’t really be a problem, as long as the sample is randomly selected from the population (which, of course, it won’t be, which is a problem). A sample of 30,000 shouldn’t give you results much different than a sample of 300 or even 30 (though your error should be lower).

    For Key Performance Indicators – which from your description seem to be metrics directly related to the advertising – what are the most popular?

    On a side note, I like to think of advertising as providing information where the cost of information is non-zero. For economists, there must be perfect information in the market for there to be perfect competition – ipso facto, lacking pure information there cannot be perfect competition; but furthermore, it means that if you don’t advertise consumers won’t know you exist.

    I’d be interested to know how much advertising is “informative” vs. “persuasive” – the former telling people the product exists, the latter persuading them they should want it. Ever since Watson left Psychology and went into advertising, advertising can – and has – been used to persuade people to do things. For instance, Maxwell and the coffee break, etc.

    And, of course, the ROI on each.

  • Andrew_Eifler

    Michael – thanks for commenting – I was hoping you were going reply here. I’ll address your points one at a time.

    As for evaluating ROI based on a “test/control” basis (i.e. seeing what sales are without advertising, then introducing advertising and seeing what happens) – this is a method that is widely used – however it tells us little about exactly which parts of our advertising efforts are most effective. For instance, if we advertise with 10 magazines we might have evidence to support the success of the advertising in general (if we see sales go up), however we would not know exactly which of the 10 magazines had the biggest impact on sales. It would certainly be possible that only 1 or 2 of them worked at all – and the rest of the magazines would be a waste of money. As a counter to this argument you could say to just test each magazine individually (perhaps each in a separate market), however that would a) take way too long for any meaningful learning, and b) the impact of any individual magazine will likely be weaker than it’s fair share of the group – as impact tends to be greater when several magazines act in concert.

    As for sample sizes, I agree with you that random selection should counter sampling biases – however I would ask you to consider the following scenario:
    A web survey is conducted to see who remembers seeing a specific banner ad. The survey is “served” to both a test and control group with the goal of 50 respondents in each group. Although true response rate is rarely reported by companies who conduct this sort of research, lets hypothesize that for every 1,000 people who are “served” the survey – 1 person fills it out (.1% response rate – which is actually high for online standards). Would you feel comfortable with the statistical validity of a survey that was refused by 999 out of 1,000 people you sent it to? Would you consider this random selection?

    KPIs are tricky – they can really be whatever you want as long as you can make a reasonable argument that they are good proxies for ROI. For online KPIs can be Click Through Rate, Interaction Rate (for expandable or interactive banners), or Website Page Views (using technology to track users through to their activities on the brand website). For offline media KPIs are usually harder to calculate – but you could, for instance, use something like Google search volume as a KPI for Print or TV advertising – with the theory that the more effective your advertising, the more people who will search for your product on Google after seeing your ad. The reason KPIs are important is because it gives us something to optimize against. For data to be actionable you need to decide what you’re going to make decisions based on – however frequently those decisions are based on KPIs that are weak representations of ROI.

    I like your perfect information argument – I definitely agree – lots of advertising is just an effort to add liquidity to the information market. Then again, as you point out, advertising can also be used to persuade and perhaps make people do things that are not necessarily in their best economic interest (I think the lottery or famous scams like “cash for gold” are good examples). In my opinion this is why the advertising industry is regulated, but as with any regulation, it’s not perfect.

  • Well, damn, I completely forgot to respond to this earlier.

    Good point about sampling. Sampling is, of course, bloody difficult.

    KPIs sound like a good example of YCMWYDM – “You Can’t Manage What You Don’t Measure”, also known as “You Only Manage What You Measure” which is slightly more cynical ;-)

    Here’s a question: How do you pick CPIs?

    The obvious way, of course, would be to measure everything possible and then perform a post-hoc factor analysis on what varied most significantly, and had some theoretical relation to the campaign. That can then provide KPIs for future (similar) campaigns – and sounds like something you’d do on the agency side, not the client side.

    One thing you touched on is what one might call the “interaction effect” of multiple channels in advertising. Disentangling that can be very difficult.

    One example is how online advertising can drive sales. If you think that online advertising will spur online sales, you’d be right – but for large discrete purchases, online advertising will *most likely* spur offline purchases. e.g. people will do research online, and then go to the store. Your ROI needs to be measured cross-channel, which can be difficult because some companies break out their retail and online divisions separately (biggest example is WalMart; their retail arms in on the East Coast, but their online division is based in San Francisco).

    The other cross-channel problem is the “cumulative” effect of advertising. If you reach a customer via magazine AND radio AND TV, is the cumulative effect larger than the effect of all three put together? (sum greater than parts, etc). I expect the answer is “yes,” and trying to figure out your spend on each channel then becomes bloody difficult.

    One thing I forgot to mention about the “perfect information” argument is that it provides an explanation for brands which, while obvious, doesn’t seem to be stressed. That is, if you have imperfect information, then a brand can act as a PROXY for product quality. So you know a Sony TV meets some minimum quality baseline, for example – you’re essentially outsourcing the job of ensuring that the product meets your needs to the brand.

    Now, that misses the personal use of brands, probably best described (in the abstract) by the sociologist Bourdieu (in his work Distinction); that is, a brand can act as a “social symbol” and convey information about the individual to other individuals. If you drive a Porsche, you know that other people will infer things about you based on that (they will attribute wealth, a taste for adventure, etc).

    Such a use is only partially under a company’s control… but it’s probably the best example of “viral” marketing, because it’s the ultimate self-replicating function. You remember those “fads” that would sweep through primarily school in grades 3-8? After the product hit critical mass, then *everyone* had to have it or else they weren’t part of the group – they were excluded – so it spurred more sales. Having the item didn’t indicate, so much, that they wanted the item so much as they wanted what the item got them (membership into a group).

    On an unrelated note, I recall reading Kathy Sierra’s blog, back before she quit (over death/rape threats) and she – talking about marketing to software developers – always asked “How will your software get the user laid?” No one REALLY cares about a new Office package, a new operating system, a new email client, a new widget to make printing flash cards easier – what they care about is “getting laid” (or whatever else they want to do). It can be as simple as saving them time, to making them look better, and so on. It was an interesting perspective.

    Anyway, the last “really interesting” topic is “How do you make sure you’re reaching the people you want to be your customers?” As you point out in your post, CTR could just target those people who tend to click on ads. I don’t think there’s a good answer to that; one response might be “Well, the data being collected by Facebook, Google, and Microsoft (among others), the fine-tuned demographic, behavioral, and (anonymized) social graph data we’ll have access to will allow extremely precise targetting. And we can even use the information we discover about them in offline targetting! (not that it’s needed, what with the move to smartphones and always-on GPS-enabled mobile devices).”

    But (i) it’s not really satisfactory, and (ii) it’s probably 5-15 years away from being the status quo.

    On a *completely* unrelated note, are you sure you want to have that be the class discussion? Because I’m pretty sure the answer is “no,” and, while the answers are likely to be cute (by which I mean “thoughtful” and “spurring thinking in the right direction”), one could say its inherent unanswerability renders it a form of mental masturbation. Sort of like (most) philosophy.

    Unless your goal is to ferry out and then crush incorrect assumptions about advertising, which would be worthwhile.

    I’m not sure there are better questions, but I’d probably suggest a few questions you could transition through, such as:
    “What do you think are the strengths of advertising? Is there anything we do really well?”
    “Where do you see the biggest opportunities for improvement?”
    “Do you think most advertising is ‘good’? What defines good advertising? Does it reduce to positive ROI? But given the difficulties of measuring ROI, can we really trust such metrics? And, surely having just a positive ROI isn’t enough – we want the *biggest possible* ROI, right?”

    The last ‘question’ is sort of multiple related questions…

    … my point being that, in my experience, impossible questions are easy for students to breeze through. Just toss out a few obvious answers, perhaps link it to something earlier you said, then admit the incompleteness of your answer and be done with it. It’s pure speculation, and not really critical thinking – there’s no examination of the existing knowledge base, particularly for inconsistencies or mistaken assumptions.

    I’m sure there are better questions than the one I listed, as well; perhaps more specific, like “Who do you target with your advertising? The people likely to buy, or the ‘influencers’ – such as the ‘mavens’ Malcolm Gladwell outlines in his (flawed) book the Tipping Point? Can you tell the difference?” But, as I’m not a marketing expert, I can’t be expected to provide good questions ;-)