The signal and the noise in advertising

Perhaps there aren’t any people within agencies who can create this model and forecast…

A marriage between creatives and quants….won’t somebody think of the children!

Nice post. It’s a few years old now but ‘The Black Swan’ by Nassim Nicholas Taleb is another great read on probability, the dangers of overestimating our predictive ability and how/why we fool ourselves into creating ‘illusory correlations’ and fall into statistical bias (which is particularly relevant to the marketing industry!)

… and never confuse correlation with causality.

Agree with Daniel’s comment. Doing this properly requires a far greater/different skillsets and resources than exist in traditional agencies. The link also needs to be broken between those that do the modelling and subsequent planning recommendations and the rebate gravy train that underwrites the existence of most media agencies.

…and yes John, never confuse correlation with causality.

JG as usual you’ve hit the nail on the head here. I totally agree we must be diligent when we look at any models so as not to boldly proclaim that cigarette lighters cause lung cancer.

We need to do a better job looking beyond individual campaign data and using our increasingly available ‘big data’ for working out which inputs are the real drivers of campaign performance and which inputs are merely passengers along for the ride. The field of econometric modelling is getting a lot better at doing that but many of the media models I have seen to date are very raw with little predictive ability. Until our media models start incorporating more granular inputs (e.g. reliable measures of creative strength plus reach and frequency measures for all forms of media as a minimum) then their value will be remain somewhat limited.

Does anyone at PHD have a PHD?

Quick disclaimer I haven’t read Nate Silver’s book and I am not a statistician, however I have a few that work for me. We undertake media modelling. The usual suspects

– Advertising effectiveness
– Future level of demand
– Importance of price

The first two are focused on prediction the last on causality. There are big issues around undertaking all of them. As already outlined the quality of the inputs are as crucial if not more so than the quality of the model itself.

With this kind of (media) modelling we also have to deal with the huge amounts of inter correlations between factors – lots happening at the same time. There is also the effect of a variable lag on both the explanators and predicators.

There also seems to be an element of art as well as science (often hidden in a black box) in the subjectivity around weighting the importance of certain factors.

To John’s point and although it may seem obvious getting someone other than the statistician to ’eye’ or ‘sense’ check the model, ideally with some understanding of what is being modelled, is invaluable.

Although I am biased this is best left to the professionals – and yes that isn’t advertising or media agencies. This isn’t a sales pitch I’m anonymous, more of a warning, I have seen outputs posing as econometric modelling from agencies and some of it (not all) was frightening.

Me again. Apologies, I also I haven’t read Silver’s book, but I have kept a close eye on his work and progress.

I also note with interest that the bedrock of his work and successful political predictions is re-analysis of what is primarily ‘traditional’ research polling. That is, the underlying research ‘data’ is not fundamentally different, but his re-analysis technique takes into account the underlying (deliberate?) biases of the US political polling to ‘recalibrate’ that data to reflect what occurs at the four-yearly polls.

As an example of what this means using local data you will notice that the political polls vary here in Australia (sometimes wildly) between the major pollsters (Morgan, Newspoll, Nielsen and Galaxy). They all use different methods, questionnaires, modes, timing and weighting to ostensibly arrive at the ‘same’ result. But even within a polling organisation there are/can be underlying differences. For example, if you look at Morgan’s polls over the past year and a half (around 60 of them), ‘mode’ appears to be a significant variable. On average, their phone polls produce an LNP result almost 3 points higher than their face-to-face polls – same questions, different answer. However, with this knowledge you can take into account questionnaire mode and produce a much more reliable prediction (as well as learning which mode returns results closer to the actual election results and reading too much into trend without understanding the mode). They key is that while both sets of data are subtly different they are BOTH usable. When you extend this across all the major pollsters you clesarly get a better picture.

Researcher echoes the point of inter-correlations in marketing analyses. The better models are generally parsimonious – that is they tend to ‘sacrifice’ slightly higher ‘fit’ to keep only the truly significant variables by discarding the auto-correlated variables. My rule of thumb is if you have more than six explanatory variables have another look at your model. Consider using a single ‘relationship variable’ rather than multiple variables. For example, you may be able to use your brand’s price relative to the market average or relative to your key competitor, rather than loading in all the prices.

Also, look for non-linearity. Very few relationships in real markets are linear (e.g. price elasticity) so if your model has lots of linear relationships and straight-line graphs in it, then that should be a warning sign.

Also, be wary of models that claim to predict “96% of all sales movements” and if they do, look for a variable called ‘unexplained’. If your model can explain sales movements 70+% of the time you’re doing well. Don’t forget that we’re dealing with real people and real consumers out there and they change their minds and buying habits all the time.

Penultimately, test and re-test your models. And when you’ve ‘got it right’ throw some more data at it and test it again and again. Withholding data from the model or doing ‘split’ models is one way of checking predictabailty.

Finally, then ignore all the stats and make sure it passes the “does this make sense” test. If it does, then show it to the client.

uggg pls refrain from using “causality”.
correlation =\= causation is the preferred expression!

Great review, and I am glad to know other people in the Advertising business are paying attention to Silver’s book — and to the topic in general. At our agency, which historically has been mostly about mass advertising, we now have a 40-person department working on all things statistical. Over the last several years they have made some impressive contributions to the work. The creative work, even.

Here is my review on the book, also an Ad Biz POV: http://admajoremblog.blogspot......noise.html

Thanks!

+1 to Mike’s comment from 9 Jan.
And Taleb’s new book highlights the folly of almost all prediction.
Whoever cracks applying Bayesian statistics to advertising outcomes will be a multi-millionaire.
Personally, I don’t think it can be reliably or robustly done.
But open to being proven wrong.

Nate Silver shows that prediction is very difficult. Many predictions are totally unreliable. Professional forecasters know their limits.

Yet brand equity firms tell us they can predict a brand’s future from just a few simple survey questions. How gullible do they think we are?

And marketing mix modellers tell us they can predict a brand’s future from just a few (now historic) correlations. How gullible do they think we are?

There’s a cynical old definition of an economist as “someone who can predict the past with 50% accuracy’. Seems about right …

Byron, while I respect your cynicism I have seen numerous econometric models which have surprising success doing just that. Any marketing mix modeller who says that they can predict EVERY brand in EVERY scenario with absolute surety is clearly a snake-oil salesman. But to dismiss every model for the sake of the few (or even the many) is equally disingenous.

I suspect a lot hinges on the definition of a brand’s “future”. My rough rule of thumb is “six for one”. If you want six months forecast I will want three years of (at least weekly) data. And in my opinion there are few brands whose historical data older than three years is still relevant.

There are various techniques which can test the predictive capabality of the model. Let’s say you have three years (156 weeks) of time-series data. Build the ‘complete’ model on that data set. Then remove the 26 most recent weeks and build the model on the remaining 130 weeks of data. First, the two models should look similar – similar brand drivers at similar levels. If not, you simply have noise and tell the client that. Second, you check the predictions from the n=130 against the actuals from the n=156. Again, if there is not an acceptable fit then tell the client that – at least they now know that their brand is either unprecitable, its category is unpredictable or they don’t have the data that underlies the key brand drivers (if indeed they do exist).

As your own book How Brands Grow shows, brands (in the main) are built on long term marketing efforts, which is consistent with the underlying premise of such modelling and data sets.

Cheers.