When I was a freshman in college, thinking about becoming an English major, the last thing on my mind was “statistical significance”…of anything.
I think the only statistic that was on my mind was how many unemployed English majors there were and whether my parents would be paying tuition for my sophomore year once they found out my plan.
It was abundantly clear that I wasn’t going to learn about statistical significance in my English classes nor was I going to learn much about it in my marketing classes either.
As I recall there was one paragraph in my “Introduction to Marketing” textbook (and it was a very short paragraph—one sentence) that mentioned “direct marketing” and anything regarding the notion that advertising and marketing could actually have a return on investment, that everything was testable and statistics and numbers in marketing mattered.
In that textbook and course I learned all about about image advertising, brand advertising and other flavors of non-measurable advertising…even how to set up displays inside the liquor store I worked at in the summers during college.
While direct marketing was not in the curriculum, I learned how to strategically place impulse items and the bottles of wine on clearance due to excess inventory in places where there was more traffic in the store.
So I had that going for me during my summer vacations from college.
But heaven forbid learning about creating any metrics to track those sales.
It was only many years later when I learned about direct marketing (outside of the classroom and the liquor store) and I got my first lesson in statistical significance and why you have to believe your numbers assuming they are numbers worth believing.
More on that in a minute.
Back to college…
During that freshman year I recall the campus radio station asked the student body to vote on “the best rock and roll songs of all time.” The year was 1977.
Those of you who ever participated in exercises like that in that era, know that number one was always Led Zeppelin’s “Stairway to Heaven” and number two was always “Layla” by Derek and the Dominos (featuring Eric Clapton).
I can’t even recall what song usually ended up a distant third.
But I was sick of the same results all the time so as my protest, I stuffed the ballot box with a song and band I liked at the time: “Running Hard” by Renaissance.
Renaissance was popular but no one was confusing them with Led Zeppelin or Eric Clapton.
And “Running Hard” wasn’t even their biggest hit…that would be “Carpet of the Sun.”
You probably never heard of the group or either song…which makes this story even more interesting.
You should check them out, though. Annie Haslam, the lead singer, has an amazing voice with a five octave range—you might mistake her for an opera singer singing rock and roll. And she still sings and rocks today (I was fortunate to see a reunion concert recently).
Anyway, I spent the two weeks while “polls were open” voting for “Running Hard” as the top rock and roll song of all time. And I voted a lot…multiple times and every day.
I assumed they would eventually ignore the votes…but they did not.
I attended a highly respected, large state university known for mediocre football and basketball teams (actually awful most years) but not known to be a school full of dummies.
However, in my freshman year, to my mind, they (at least the students running the radio station) became known for not understanding a result that needs to be thrown out and a result that was not statistically significant.
Even this English major could have figured that one out.
Little did I know, I was getting my first lesson in statistical significance—one that was never going to happen in a classroom as mentioned earlier—and one that I didn’t learn in earnest until meeting Dick Benson a few years into my direct marketing career.
So what did my college radio station do?
When they counted down, the number three song of all time was “Layla,” number two was “Stairway to Heaven” and number one was “Running Hard.”
I remember telling my friends that I felt really good about my college choice with so many Renaissance fans in the student body. What a pleasant surprise! 🙂
Fast forward to the first online launch I ever did (sometime around 2008 or so).
I recall we were testing two offers in real time, online…which was very exciting since the usual for my company up until then (offline) was that we needed to wait 12 weeks to get actionable results while doing split testing in direct mail.
I remember we were testing a particular variable in the copy and a few hours into the test, we had 12 orders for one offer and 8 for the other which caused our launch expert to exclaim:
“We’ve got a winner!”
This was on a $97 offer mind you…there was still more time to run the test (in addition to the numbers being so small)…so of course I would not let our expert roll out with his proclaimed “winner.”
And at the end, the test ended up 23 to 20 orders (or something like that) and there was no clear winner or any kind of statistical significance.
I used these two extreme examples to emphasize the need to be educated in how to test effectively and accurately in direct marketing…something we could take a deep dive on here…but I will wait to share that deep dive in a future post with an excerpt from my new book on this critical topic.
Some broad things I want to share today though:
The first rule of thumb I learned about statistical significance was when Dick Benson encouraged me to have at least 100 net orders (in our case on a $30 product) on each side of an A/B split test as a starting point to have confidence in the results…and that rule can get extrapolated in a variety of ways, online and offline.
(Note: The science of a true “A/B list test”–making sure the two groups are alike in terms of their list profile—is also a key thing to understand and make sure you can rely on your test results. You need statistical significance on the responses and you also need list universes that are demographically similar. More on that when we deep dive).
The other rule of thumb is to have an analyst (with an expertise in statistics) as your right hand person if you are a marketer—and it should be an analyst who is not an “English major type” but a “math major type.”
I recall a guest speaker at one of my Titans Mastermind meetings who had built a company that reached nearly $100 million in revenue at its height–and he said that when he launched his business he assumed his most critical employee/internal partner would be his Chief Financial Officer.
However, after building that almost nine figure business on the principles of direct response, he said the most important employee in the company was his “Director of Analytics” since he learned that without accurate numbers on the true profitability of every campaign (based on statistically significant, measurable results) the company would have faltered many times. This was the key to their growth.
He is not the only CEO/business owner/entrepreneur who figured this out.
And it is always the direct marketers who figure it out first…and best.
I just received an amazing endorsement for my new book, Overdeliver, from direct response marketing icon Dan Kennedy who said this about the book (and direct response marketing):
“Pros will find it fascinating. Those new to the disciplines will find a vital crash course. Pin-head executives in big, dumb corporations wasting oceans of money on utterly unaccountable brand and image and ego advertising should be forced at gunpoint to read it.
Ogilvy was right when he ranted to his own agency’s staff that only the mail-order people knew what the hell they were doing, and were Ogilvy alive, he would applaud Brian’s work here”
Of course Dan almost made me cry saying David Ogilvy would like my book.
But that is not why I shared his observation with you–nor am I bragging.
I am sharing it to give credence to what I am talking about today.
The lessons are not just for “big dumb corporations” who don’t get it.
It’s also possible that there are still some college radio stations and online marketers who don’t get it either…and all of us need to understand direct marketing and the principle of statistical significance.
The examples and people I have cited above might not lose “oceans of money” due to their bonehead thinking.
But I would recommend that the radio station should only be looking for the number three song of all time when they ask folks to vote again; and the online marketers mentioned above, and those like them, should hire a kick ass marketing analyst as a top priority, especially if math and statistics are not their strength.
I know there are many pieces of software that can help with this important function but it’s a role that deserves a lot of personal attention too.
In short, they (and you) should do what David Ogilvy, my new BFF, says about that:
“If you ever find a man who is better than you are—hire him. If necessary pay him more than you pay yourself.”
Warmly,
Brian
P.S. All this Ogilvy talk reminded me about the classic swipes and videos that are part of the resource page which are all free for buyers of my first book, The Advertising Solution.
There are ads which should be part of every marketer’s library and there’s even a video of Ogilvy on “The David Letterman Show” which is priceless.
If you already own the book and have accessed the resource page, let me know what you think.
If you have bought the book and not accessed the resource page, please go to the URL listed in the back of the book.
And if you haven’t bought the book yet, buy it at www.TheLegendsBook.com and follow the instructions on that page so you can get the bonuses once you purchase.
I make nothing on the sale of the book and I guarantee it will be the best $16 you will ever spend (that’s what the book goes for on Amazon).
Each of the 6 legends profiled in the book, Ogilvy being one of them, are “direct marketers trapped in general advertiser’s bodies.”
And all of their best thinking and rules of thumb are in the book.
I believe we owe these pioneers a debt of gratitude because they trained us to make our advertising measurable, accountable and always statistically significant.