The Mumbrella 360 Experiment: stop talking, start (them) acting

Adam, very interesting post and a highly commendable experiment. Extracting any money from tight-fisted media folk is worthy of an EFFIE in my mind 🙂

The only comment I have to make is that I don’t think you are actually ‘flipping’ AIDA, but instead you are cleverly accelerating it – more of a “Rapid” AIDA if you like …

I know “Flipping AIDA” is a better soundbite, but it suggests people will act without any awareness, interest or desire of something … do you think that’s true?

Out of curiosity, how much did the control group donate? Does completing a puzzle increase the likelihood of donating money?

Hi Adam, I agree ‘flipping AIDA’ is a nice sound-bite and one of the reasons we like the term. However, we also think there is a lot of truth to it. As long as the action the person undertakes is aligned with the ultimate behaviour change you want to effect then flipping stands.

Hi Nick, very good question. We deliberated for a long time on what the control group should do. They can’t sit there and do nothing, and they had to do something, so we thought unrelated puzzles was the best thing (we took advice from Deakin on this too).

Sorry should have put the amount they raised in the article – it was $2.58 each

Times must be tough if a room full of media types only have $12 cash in their pockets.

A very interesting experiment Adam, and very interesting results.

Could there however be a ‘self-selection’ bias here? The third group was asked to create an advertising campaign for the charity. And what was the primary composition of the audience – advertising people.

What would be really interesting would be a replicate study at another conference – say a bankers conference. Should the same result be achieved then I would have to say I would totally agree with your conclusions, until then, the jury is still out for me.

An interesting exercise.

I would suggest that “forced ADIA” (note reversal) would be closer than flipped AIDA. You’ve told these subjects, in front of their peers, to do something regarding charity. Walking on by was not an option – you’ve forced the awareness, interest and desire.

This leaves a gap in getting people engaged with the “crusade” / “cause” in the first place out in the real world. This is where Rational Choice Theory can foil the AID in flipped AIDA.

As the ad-making group have also invested their time and personal creativity, they are likely to be subject to Social Exchange theory – I have skin in this game, so I should support my effort. It’s like why club members are usually the strongest donors. It’s also why people are encouraged to make banners for marches.

What this does highlight is the emerging strength of “crusade marketing” (where the personal reward is meaningful participation) over “cause marketing” (where the reward is transactional conscience salving) – GetUp! would be a salient example of crusade marketing strategy.

It’s a while since I’ve done this stuff in anger, so I hope this makes sense.

@john Grono Nothing in science can be proven, only disproven. We have developed a hypothesis and are getting a fair wack of evidence to support it – but as you say there will be people who, quite fairly, doubt it. So replicating the experiment with another group of people would be fantastic. Also, we toyed around with other action orientated ideas but thought the write an ad one would be fun. Thanks for your thoughts.

Adam, agree with the proven/disproven comment – hence null hypotheses. If the replication exercise becomes a possibility I’d be happy to assist in any way that may help.

Really like the thinking and I like how it appears Adam is happy to be proven wrong. Appetite for risk and failure should be celebrated not criticised.

Hi Adam,

Good experiment on willing bunnies – make them involved and they will therefore defend the cause as they will find it very difficult to refute or invalidate themselves.

Question i have is the use of the term ‘cognitive dissonance’ to this behaviour. My understanding of cognitive dissonance is around the conflict one has once one understands the impact of (impulsive) behaviour and how you attempt to justify it. It explains the motivational desire to reduce the conflict or dissonance between possible thoughts.

Such as….I just spent heaps/more that necessary on a new [car, suit, dress, bag] but it’s justifiable it because it [will last, is quality v quantity, i deserve to bask, it’s safety etc etc] but the feeling of uncomfortably never really dissipates – it just becomes more justified (!).

Ergo, the commercial role of communication is therefore to both entice (AIDA) and provide the stories the consumer can use when encountering cognitive dissonance – like Mercedes does with safety and Moet does with celebrations, nike does with winning etc etc.

My read would have been to refer to it as ‘effort justification’, where the thoughts you may hold about a subject matter (does that charity really need the money, couldn’t the govt do this, Mercedes are soooo expensive, who really needs French champers etc) are turned on their head by involvement causing one to change views and advocate, (the world needs this, my family will be safe, i just won a big pitch and feel like a winning spots star!).

Great live experiment…your thoughts?

@adgrunt makes sense absolutely. and I agree it’s equally plausable to say ‘Forced AIDA’

@John thanks.

@logic again thanks. Happy to be proven wrong – absolutely. Just hope we’re not!

@Andrew Duckmanton we call what you’re referring to as ‘The barroom defence’ the requirement to have a rational justification in your head for an emotive purchase (to be blunt). This, I understand it is an attempt at minimising cognitive dissonance. Our point in the experiment is that get people to act first and they have to minimise CD by changing thoughts and feelings as the behaviour cant be undone. In terms of doing the live experiments – go for it.

I just had the horrible realisation that the “absurd” end of my hypothesis is that ad competitions for charities are possibly the way forward for increased donation (esp amongst donor base).

Noooooooooooooooo. Though it might work…

Hi Adam,

Would love to know what the ad the third group came up with was

what if it is a new charity? does awareness have to proceed action? I think so, in fact, I know so. but a great PR exercise for naked…congrats.

@adgrunt yep, consumer generated content (consumers writing ads) is unfortunately here to stay.

@adverb there were a few nothing amazing, but some really nice thoughts including making a paper chain of children

@jim i think you’ve missed the point. But thanks for contributing.

In conferences, yes. 😉

I’m a huge fan of using research beyond traditional practice, so well done Naked.

However, having been in the 360 audience during one of your trials, I couldn’t help but wonder how much response was just public guilt, and how much was a reaction to the stimulus provided.

Passing a plastic bucket in public along rows of people which creates a loud “clunk” every time a coin hit must surely have had a massive influence on shaming people into donating – irrespective of what they had just seen, heard or participated in.

I was bored shitless by the stimulus I had to sit through, but I still dropped a coin in to avoid public humiliation. In a real environment I wouldn’t have noticed the stimulus in the first place as it was so boring.

As is with all forms of research, the rational group environment in which the stimulus is tested in is about a million miles away from the real world context in which ideas are really judged and split second emotional buying decisions are actually made.

Having sat behind two-way glass observing focus groups for over 20 years in 5 countries leads me to believe that you can no more predict the effectiveness of a TV script than you can predict an earthquake. Especially a script that involves dialogue & performance.

Nobody buys anything for a rational reason, so having a 3 hour rational debate about a raw script in a room full of strangers where nobody really understands what’s going on, or tells the truth, is a bit of an expensive con job.

YouTube used to have the numbers 1-5 for viewers to give feedback on how much they liked or disliked a clip. Now they just have a thumbs up or a thumbs down.

That’s how fast it works in the real world.

So you can spend as much money as you like on as many focus groups as you want and read as many thick reports as you like full of adspeak & pie charts, but it’s all rubbish.

Focus group testing has as much scientific credibility as Intelligent Design.

All you need to do is make Taylah from Toongabbie smile, and you’ve made a sale.

Yo. Howard from Save the Children here. Only one way of knowing if this experiment demonstrates real social change. Only one way of knowing if we’re flipping or accelerating AIDA. Adam: time to go public!

I’m a fan of all the new thinking relating to how marketers can actually drive meaningful behaviour change. It’s an exciting time to be in the industry as we move from assumption to evidence, and I’m all for trying to flip AIDA on its head… ‘cause I think it’s mostly rubbish, too.

(It needs to be noted, however, that this “new approach” is not all that new. Old school admen like Lee Clow and Bob Hoffman have been saying the following for years: We don’t get people to try our product by getting them to love our brand; we get them to love our brand by getting them to try our product. In other words, changing behaviour *has* to come first.)

Further, I applaud your (and Naked’s) gadfly status in the industry. God knows there aren’t enough of you down here.

But here comes the “but”…

The Ad Nerd in me says: Your experiment proved the hypothesis you set out to prove. Given the ‘decline effect’ – http://www.newyorker.com/repor.....act_lehrer – are you worried about this? If not, why not?

Further, most of us at m360 knew we were part of a Naked experiment – we knew were being observed. People who are aware they’re being observed modify their behaviour. And those conducting the experiment often unconsciously modify *their* behaviour to drive the result they’re looking for. (Hence the ‘decline effect’.) For example, those Naked-ites running the experiment in the room making ads *knew in advance* that was the room you wanted to win, and may have unconsciously behaved in a way that drove donations. This might explain why that room donated the most, no? (I know you weren’t in that room to observe whether that occurred… because I was.) All of this is intended to make this point: These kinds of experiments rarely capture the “truth”.

(It is also the reason almost every focus group is a crock of shit.)

Further still, several of the psychology experiments from the 1960s and 70s you reference in your talks have been discredited in the intervening years, mostly because of the “observer effect”: the participants behaved in line with what they thought the people watching them wanted them to do. This means that what respondents did in the lab rarely syncs up with behaviour in the real world. (Chapter three of Superfreakonomics is instructive in this area.)

Why am I writing this? It’s not to take a shot at you Adam. You are one of too few people in this industry pushing things forward, which I continue to respect.

But my point is this: I agree with the model that getting people to take action is the best way to drive meaningful and lasting behaviour change. However, dressing it in the language and garb of “science” and “truth” is a step too far for me. We’re simply not there yet. The experiments conducted and referenced, and the advertising case studies mentioned simply aren’t robust enough to say this new approach is “true”. (Clicking a “like” button isn’t meaningful action that is going to result in lasting behaviour change. Let’s not bullshit ourselves.)

I guess the challenge to all of us in this wonderfully bizarre industry is to keep pushing in this direction, and develop more and better examples that better demonstrate how changing behaviour first is the most effective way to change people’s minds and preferences.

(OK… where’s the bourbon…)

@ Adam Hunt,

Can’t be arsed writing a clever response [which is fitting], but you’re bang on the money.

Bravo.

And after Hunt’s and Cummin’s talks at m360, I’m all for transparency: I’m Tom Donald from The Works.

(But I like tattoos more than ads…)

Hi Adam Hunt
If you’re interested in understanding research (and insights) beyond focus groups you should read up on experimental design that will explain things like the fact that all the things you said contributed to people donating (rattling tim, money dropping in can peer pressure etc) are all true, but the same across all groups. The only thing that changed is the communications method (the independent variable). When constructing an experiment you control for as many of these factors as you can. We think we did a good job – but no experimental design is perfect (especially in the applied setting) so it’s absolutely open to improvement – but the points you’ve raised are all accounted for.

Secondly, I agree with you on your point on focus groups (which our experiment had nothing to do with but you’ve bleed the two things together so I feel compelled to respond) to a point. Most are terrible, but not all. Many are done for the wrong reason, but not all. Many don’t help the creative process, but not all. To say ‘Research. bad. creativity. good.” is overly simplistic. Refine the argument and attempt to come up with solutions would be ace.

Also, I think your comment of ‘no one buys nothing for rational reasons’ is unusual. I bought a pie earlier today because I was hungry.

Also I think the comment of all you got to do is make tanya from toongabbie smile and you’ve got a sale might be right (to a point – likeability is correlated with sales) however, it doesn’t = build a brand. The hard bit is getting people to buy stuff whilst building a brand – that requires more than a ‘smile’.

Thanks for contributing Adam – I’d have liked to have agreed with you more. Also it’s Sunday!

Adam to Adam!

If you’ve got a sec (between baiting dodgy dick doctors & silly scientologists… which I think is utterly brilliant) please give me a link to your favourite experimental design information source. I’d like to know more but I’m as “time poor” as a boring ad brief.

Sorry about linking my “focus groups fuck ideas” rant to my observations on your experiment – I probably should have put a “& another thing” subhead in before I let fly.

I agree with you that “Research.Bad.Creativity.Good” is way too simplistic.

Research that uncovers an insight about who you’re trying to have a conversation with is a rare & beautiful thing.

Research being used to determine fonts and bizarre “frame by frame objectives” is where the whole bloated process goes into the realm of surreal scientific stupidity.

When I say that “nobody buys anything for a rational reason” I’m referring to purchases like new cars, Rogaine, Stilettos, cigarettes, politicians etc…

My larger point is that research is a failed attempt to box the passionate madness of ideas into a cubicle where adspeak robots poke, prod and attempt to predict an unpredictable outcome. And make a lot of money while they do it.

They’re just a conga-line of snake oil salesmen that drag ideas down a dark path and strangle them into the boring shit that bombards us daily.

Anyway Adam… back to your pie – what sort of rat coffin was it?

Did you know that most chunks of meat in pies are the lips & arseholes of what’s left over from the processed carcass?

That was always an amusing riposte in my mind when I suffered through that bogan ad where the tradies look at the suit in the restaurant as they tuck into their pies & opine:

“D’ya reckon they know what they’re missin?”

Lips mate…& arseholes.

Anyway, making Taylah from Toongabbie smile is a prerequisite to any positive feelings she may have about a brand down the track.

And therein lies the problem – most brand managers don’t have the time or the desire to invest the time necessary to build a brand.

Most brands are like practitioners of speed dating – you’ve just sat down for lunch & they immediately go the tongue baguette. What you need is a nice conversation first, and that takes time, and people won’t want to have that conversation with you unless you make ’em smile first.

Ferrier wins over Hunt

This is a fascinating outcome. I would add that by working on a campaign, you come to understand the issues in a deeper way because of the amount of research/thought that goes into coming up with the creative ideas.

I wonder how this principle could be applied to chuggers? They drive me crazy!

So, are initiatives such as Movember, World’s Greatest Shave and Clean up Australia Day the sort of action-based campaigns that we are talking about here? ie: Where participation is enlisted before the event and behaviour changes invariably follow? Or am I missing the point completely?

Why do some creatives think they’re geniuses? It’s only advertising, not atomic fission.

@tom I diagree – there is loads of science just waiting to be picked up and applied – and that’s what we are doing. We’re not dressing anything up in science language – we’re borrowing from it. And we’ll get better at it the more we prioritise it (as a few good agencies (creative and media) are beginning to do).

@em thanks

@Alison F you got it in one.

@Sven in all my time in marketing communications the smartest, most insightful, and interesting people are very often the creative directors (people like Adam Hunt). And thanks for your support and POV.

Since when did the Mumbrella comments get this academic (and polite)? Brilliant discussion here, so thanks to all who took the time to contribute – let’s see more of it!

Oh Adam Hunt, I can’t just sit by and watch you call us snake oil salesmen. I’m a researcher who runs your much-loathed focus groups – actually a phrase I loathe just as much for the american research type associations it entails as opposed to actual discussions that I prefer (UK/European style).

I completely agree with you about the ‘beautiful’ research that uncovers an insight vs the ‘Research being used to determine fonts and frame by frame objectives”. It saddens me that you seem to think the former is so rare. As a researcher, my interest is to work on jobs that allow this to unfold, and would do as much as possible to avoid getting involved in the latter. Ultimately the client would be the one that drives taking it down that path, we as researchers then have the choice to take it or pass on to someone less discerning.

Its why I always far prefer being involved in the early days of providing input for the strategy and creative brief rather than comparing and assessing the creative ideas/executions. My ideal group would have a one page discussion guide (if that), to allow us to genuinely discuss, rather than to make sure we get through a hundred different questions that various members of the team may have seen fit to throw in simply because they can/want to show they are paying attention.

FWIW I detest boxing anyone or anything and fortunately my clients seem very happy with me to date.

Back to the topic..

Adam F, interesting experiment, sorry I couldn’t make it..I certainly know that I’ve always said all a brand needs to get me to buy {more of} their products is get me to do a project on it. My level of suggestibility is that ridiculously high. That said, it works for Jaffa Cakes but not say, cigarettes.

Not sure how this demonstrates participation is the “the most effective way for a charity to raise money”?

All you’ve done is shown that once engaged those that “participate” donate more.

The most important element could well be gaining engagement in the first place.

If the idea of “participating” it is less engaging than an “emotional” or “rational” message then the conclusion you’ve drawn from this experiment is flawed given engagement was mandatory.

Charities could potentially drive a better outcome despite smaller donations if an “emotional” or “rational” engaged more people than a “participation” message.

In your experiment, the difference between donations from those that were “participants” and those that were engaged “emotionally” was only 9%. A margin easily rendered insignificant by a “rational” or “emotional” message engaging 15% or 20% more people at the outset.

Your experiment suggests that those that participate may donate more, but it doesn’t demosntrate that participation is “the most effective way for a charity to raise money” given engagement was mandatory. Engagement could well have a greater impact on donations and therore drive better outcomes for charities.

Hi Nicky –

I’m sorry if you took my “snake oil salesmen” comment to heart, but it’s the exasperated sigh of someone who’s been repeatedly hammered by the blunt instrument of ridiculous research methodology for way too many years.

I can only speak freely now because I no longer have to keep a straight face in front of clients during a research debrief that sees a simple idea that would have elicited a smile transmogrified into recitation of the brief masquerading as dialogue that would bore the shit out of a constipated elephant.

Now I’m NOT of the school that thinks that if only there was no research, brilliant ideas from genius creatives would rule the universe – because lazy hack creatives are as responsible as any other factor for the avalanche of atrocious work out there.

It’s early days in that orgy of self congratulation at Cannes, but it’s hardly looking like a vintage year for Australian ideas – so there are many more villains than just research responsible.

But outdated research methodology – especially focus groups spending hours rationally discussing storyboards of TV scripts are largely responsible for our daily boring shit bombardment.

My hero Bill Bernbach said: “How do you judge an idea from a storyboard? How do you storyboard a smile?”

That’s a beautiful insight that was ignored in the 50’s, and still ignored now, because focus group testing of storyboards has remained basically unchanged since then.

Equally unchanged is the structure of most BDA’s “Big Dumb Agencies” in the words of my living hero – George Parker.

But here’s a couple of local examples of ideas that were lucky enough to escape focus groups – “Not Happy Jan” and “Rabbits”.

I was speaking to Sarah Barclay (‘Not Happy Jan’ creative) the other day & she confirmed what I’d long suspected – that it was never researched. I can say with utmost belief that had there been a storyboard frame drawn with a woman sticking her head out a window & saying “Not happy Jan” being read to a focus group of strangers by a bloke in a bad suit with halitosis the idea would have died a death at the hands of Taylah from Toongabbie. The classic comic moment preceding the delivery of the line: “NOT HAPPY JAN!” was pure chance, as the window was actually jammed. You can’t storyboard that stuff.

Equally so the Rabbits ad – if I had a million bucks I’d bet it on that script dying at focus group stage – because like “Not Happy Jan” the “Rabbits” ad relied on performance & dialogue that that can only come from great talent & great direction.
(Rabbits was also helped by having a great brief with a simple insight which led to a great script, a great client who wanted great work & judged the script on gut feeling rather than research, and an agency where the principals were focused on facilitating great work on that piece of business. It takes a team)

Anyway Nicky – I applaud your desire to keep things to a single page – because any brief that is longer than a page has too many communication objectives to get across no matter what the execution.

Lazy briefs chock full of unintelligible adspeak are the primary villain responsible for the 3000 boring ads we see daily… but maybe I’ll save that for a later rant, because looking out the window I can see some kegs of Asahi being unloaded, so I better get them safely behind my bar.

Oh @Adam Ferrier: I forgot to say earlier that surely your decision to buy a meat pie was based on gut feeling?

Adam & Nicky.

I too am a researcher but I specialise in quantitative audience measurement rather than the sort of research that Adam is railing against. I suppose it is just a different type of snake-oil … hehehe.

As you BOTH point out, bad/lazy research is as damaging as bad/lazy creative and bad/lazy briefs.

I remember when I first went agency-side and there was a creative railing against the client researching his ad concept/storyboards. A particularly insightful suit said something along the lines of … if the client has decided to take it to research then you haven’t cracked the nut so it probably deserves to die (and in general that sort of research WILL kill a nascent ad). I never forgot those wise words.

Dear John –

here’s some more wise words from my hero Mr Bernbach:

“I warn you against believing that advertising is a science”

And:

“Logic & over analysis can immobilize & sterilize an idea.
It’s like love – the more you analyze it the faster it disappears.”

And this piece of poetry:

“We are so busy measuring public opinion we forget we can mold it.
We’re so busy listening to statistics we forget we can create them.”

I’m not sure I ever came across a client who took a script to research because they weren’t sure of the idea – research was always an inbuilt, non-negotiable part of the treacherous path that ideas had to tread.

In New York I spent more money making animatics for research than I’d spend on the entire production of ad here.

In my opinion what deserves to die is the outdated & ridiculous methodology of focus group testing of ideas that rely on talent & direction to work their beautiful mind magic to elicit that elusive inner smile.

Pretty hard to accurately test at animatic or storyboard level, punters just can’t put it all together. Focus Groups like this should be nothing more than disaster checks – if everyone in the room starts vomiting and says it’s absolute rubbish, then go back and try again. But don’t ask them what to change, they don’t know the answers. 15 minutes and you’re done.

I’m more a fan of quantitative post tests – test it in the wild, crunch the numbers, learn from success or failure. Rabbits was totally off the charts in post tests. Many (many) other Telstra ads were dead set junk.

Well done Adam F. Enjoyed your presentation at Mumbrella360 and concise summary above. Think the research results were somewhat boarderline, yet the message you articulated was clear, profound and thought provoking. And, illustrated the value of well constructed research. Love the focus on ACTION, yet tend to view this as just a deeper version of EMOTION.

The thing I’m most impressed with is the degree of debate and above. As a market researcher, I’m always curious at the degree of debate and polarisation the topic can shake out.

Like your conciseness @Alison F

When I have time, I’ll read the monologues of some above in detail. How do you find the time? I’m very envious.

Ps. I’m still curious about how you could apply your model to stopping smoking.

Speaking as a client who uses qual as a disaster check, I think finzen has nailed it. A little bit of research saved my bacon a couple of times. Not from dud ideas but from using certain words that we all thought were no brainers but which consumers no longer trusted. This was later confirmed in quant. The decision to go to research had nothing to do with agency or idea but was simply a matter of commonsense.

I can’t for the life of me understand why some people don’t think that running something by your target audience is a bad idea. Construct it as best you can, cognizant of the shortcomings of each approach, and take from it what makes sense in light of experience and intuition.

@ Other andrew – nicely said
@nicky its the hawthorn effect
@stringer i largely agree. effect size wasn’t massive. this experiment was in no way conclusive – just offer supporting evidence
@adam hunt and @john grono i think quant pre-testing is a hidden science and thats not good science. the big research agencies rarely share the work that makes their base line measures, how do you storyboard a smile, and a cartoon is not the same sa a finished ad.
@jason Dunstone i still don’t know!
@Sven agree that can be a very good use of research – nailing the little insights / language.

@Adam Hunt, you have beautifully expressed and laid bare all that is wrong with ad testing via focus groups. All clients in the house, please take note.

@Kevin he does have a way with words

Hey thanks for the compliments gentlemen –
but if you like my words you’ll love my Espresso Martini.
I had an idea that if I made a really good one, people might actually buy it.
So I conducted extensive background research about the sorts of things that people like about Espresso Martinis. I also looked at how other people made them. Then I started making them myself, judging them not by their individual components, but by which one gave me a simple, warm inner smile.
I didn’t want to over analyze things, because you can read as many recipes as you want, but the only way to judge the taste of an Espresso Martini is to actually make one & drink it : )