Besides the novelty factor (which others have mentioned), his 'paint' version is also much simpler, and therefor much easier to understand at a quick glance.

  - Large text
  - Not very much text
  - A single image
  - A single visual flow (top to bottom)
  - A concise color palette (greens and black)

Compare to the 'standard' ad, where:

  - Text is too small
  - Too much text
  - Too many images competing for attention
  - Muddled visual flow

Good lesson on the value of in-market testing. That said, would love to see some theory and analysis about why the MS Paint ad outperformed the standard ad. My hunch is that the first ad -- while it obviously looks a lot more professional -- looks like every other banner ad on the internet. It reeks of ad-ness, and it may set off some psychological barrier to receptivity amongst viewers precisely because their brains have been trained to filter out ads. (Banner blindness, as one of the other posters has pointed out).

Conversely, the MS Paint ad is, if nothing else, novel. It looks pretty different from most display ads out there. It catches the brain's attention, rather than being caught in the brain's passive ad-filtering heuristics. This may be, if nothing else, a story about attention and awareness.

Interesting, but the author is missing a critical point: the conversion after clicks. When I see the first ad, I clearly understand immediately that it's for a car game. I'm not likely to click, but if I click, I know what I want, and I'm very likely to actually download/install/play the game.

When I see the second ad, I wonder what it is, so yes, I'm more likely to click on it. But I'm not interested in car games. So I'm also much more likely to press the back button as soon as I understand what I'm being sold.

How does the math play out in the end and which ad is betetr? There's no way to tell from that data.

I think the comments in this thread about ctr vs actual conversions are a bit silly.

Pof's target audience here is "people who like video games" and they're trying to get people to play a free online game. They don't really have anything to teach people in verticals where the consumer is making a measurable commitment. You do not need to be very persuasive to "convert" with that crowd.

Almost all marketing advice more specific than "the customer is always right" and "sex sells" etc. is dependent on the product and the audience.

What this is is a great ad. It's really funny (The speed lines especially. This car does not look very fast, but the speed lines show it's got a lot of heart), and it understands its audience. I think it has broader application than many are giving it credit for, it probably would work with any sincere product (i.e. selling something other than "one simple rule") targeted at younger people that does not solve a "serious" problem. Once you are proposing that people spend more than $40-$50 you are pretty smarmy if you are trying to push them into an impulse purchase

No one should decide on a health insurer based on a MS Paint ad, no matter how hip, nostalgic, and casual they are. On the other hand, you can make a lot of money selling funny ads to people and I think that's what's going on here.

I suspect this is all about novelty.

Users haven't seen an ad like this one a thousand times before. Some might not even think this is an ad. Some will be curious enough to actually click.

The MS Paint banner ad is not inherently more effective. If every banner ad on the internet was hand-drawn in MS Paint in 5 minutes, the joke would quickly grow old, and the CTR would vanish.

The "test everything" mantra sounds good, but in practice, you generally have only so much data you can afford (in impressions per day, or whatever), and when your CTRs are often 0.1% or lower, you need quite a lot of data to get narrow confidence intervals around your CTRs. Using the basic binomial model, if you have two test conditions, one of which actually does 20-25% better than the other, (say, 0.11% versus 0.09%), your confidence intervals will keep overlapping until you have OOM 1M impressions. This is all just to say that running a whole lot of tests can quickly become expensive an impractical.

While testing some radical, weird treatments can give you valuable perspective, or shed light on the assumptions you've been making, testing every idea is rarely feasible. I would not, for instance, guess that that the author should test different versions of the second ad with colors or number of exclamation points changed.

The problem is, that this only works because it's so completely different from all the other ads and because of that reason escapes some of the banner blindness.

We've had similar results when we modified our logo and added mistakes. For example a rectangle Google AdSense Banner with a mirrored version of our Logo or just some crazy saturation affect applied on top of it actually got excellent CTRs. Even though it was kind of unnerving to have all my friends tell me about the mistakes that were in our ads...

Actually advertisers have known this since a long time. Ugly wins more attention. More attention = higher conversion.

The first job of ads is winning attention. You have to fight against all the clutter and stand out. You can do so by a variety of tactics. Use human faces. Use cute looking models. Use ugly fonts or clipart. Use mouse pointers. Use fluorescent colors. Use dashed coupon type borders.

You need to know however that while your conversion rates will increase, the number of complaints you receive will increase too.

Possibly one of the most exaggerated headlines I've ever seen, and that's saying something.

This is actually well known among savvy internet marketers. And it's why those tips for belly/whitening/wrinkles "invented by a local mom" always look far from professional.

It works because:

- It stands out from the content

- It defeats ad-blindness

- It's not expected, so it makes you curious

If this style becomes the norm, users will learn to mentally filter ads made in Paint as well.

I think the first ad says "I'm just another giant corporation who wants your money."

The second ad says "I'm a human being, probably with a sense of humor"

FIRST: tylerrooney, thanks for posting my blog on Hacker News! Much appreciated. Judging from the comments, we've got some super knowledgeable people here.

So to clarify, this was simply a CTR case study, I know there's another side of the coin for CVR but that would have taken more time and funds than I would have been willing to allocate :(

But hey, open invitation the community here: If you want to submit a 310x110 ad for the purpose of testing against the same demographic that I did, feel free to email it to me: ben@pof.com. I'll run it for a few days and I'll let you know how well your ad did :) And to make it worth your while, highest CTR ad (from Hacker News members ONLY) gets $100 credit to advertise on https://ads.pof.com. AND you can use your affiliate links so if you make some coin, it's yours to keep (Put it towards your Diablo 3 pre-purchase, yeah? lol) Just grab the direct link for Need For Speed World from your favorite network and send it over. And hell, if your ad beats mine, I'll post it on the POF blog, with your permission of course.

End date for this little challenge.... April 30th?

This is hilarious, but it doesn't take into account one of the major factors of online advertising, and that is branding, or brand awareness. While the official EA ad may not get as many clickthroughs, what it will get is subconcious eyeballs, and given enough impressions of the same ad in various formats, same EA logo, car, game title and branding, you don't need people to click on the ad to start to recognize that there is a new Need for Speed game available. It's similar to flooding the airwaves with a particular TV ad. The goal is awareness, which over time, can lead to a purchase, whether the person decides at some point to purchase online or offline. Maybe they're at GameStop and EA has placed an in-store display with the same EA Need for Speed branding. The person may have forgotten about the game, but walking in to this GameStop, they're memory is refreshed of it by being previously exposed to it through digital or television.

So in short, what's more important? Immediate click-through satisfaction or building real brand recognition that can show greater returns over time, mostly in ways not calculable?

I work with many MBA colleagues and whenever they encounter an ugly, but effective (ie. converting) ad, they can never drop their egos to accept that such an ad can work. They just feel they cannot "stoop that low" to adopt these techniques.

They would often blindly push for "simplicity", "sparse text", "nice picture", and when these ads go out to market, they get absolutely crushed.

One of the tragedies of a big wealthy company is that marketers can continually go out with these crappy creatives that don't sell and there really is no big consequence. It is often written off as a "learning opportunity".

Whereas if you look at the ads of people whose lifeblood depends on selling their product, they may not be the most attractive ads, but the ads that persist over time tend to be effective (ie. they sell). These guys need to eat, and they can only afford to make stupid ineffective ads for so long before they starve! So there are definitely some practical lessons that can be learned from them. They often knowingly or unknowingly follow the principles of advertising legend David Ogilvy.

Their ads tend to hit on direct marketing best practices:

- headline states in plain language what the product does (ie. no MBA jargon/buzz words)

- headline also hooks the reader to read a bit more

- it is clear who the product is for

- copy combats any objections in reader's mind

- no distractions that divert reader from clicking the "Join Now" button

- contains customer testimonials reader can relate to (ie. social proof)

- gives reader enough information on the page to make a decision (ie. none of this sparse text BS if it doesn't make sense)

Is it just me or did anyone else find the first ad terrible and the second one quite good? I think I have been in the game too long.

One of my favorite ads of all time was the stick drawn fat girl with the secret to losing weight. I am sure the person behind that ad has a serious bank account.

I love this kind of social experiment. There are a few theories as to why this resulted in a higher CTR. The obvious ones to me are:

1) "Look at the pretty picture!!"

2) "What the hell? EA is allowing this ad? Did they make it? I gotta see where this goes!"

3) Alternate to "Banner blindness", as @jonnathanson pointed out.

In 1 & 2, I feel like the higher CTR wouldn't matter because people are acting on curiosity of the implementation rather than the product. Once the outcome was revealed, and the banner is seen as no more than a trick, I'd be willing to bet that the orders or pre-orders of the game (in this case, playing for free) stayed roughly the same as if using the other banner. Just a hunch.

If you guys think this ad is on to something, you really should see the ads put up in the SomethingAwful Forums (general subforum: http://forums.somethingawful.com/forumdisplay.php?forumid=1 ). The interesting thing is that many (I'd estimate more than half) of the ads are put up by forum members themselves, making for a range of very informal, hilarious, parodic, often obscene ads that play to every stereotype ironically and knowingly, sometimes made shittily with MSPaint.

And where do those ads lead? Usually to a forum thread where members are playing/raiding an MMO together, or discussing a topic of great interest, or selling a service like painting portraits, web hosting, or resume editing (and in one or two cases, to a discussion of a particularly zeal-inspiring anime series).

Some purely parodic examples can be found here:

http://acapella.harmony-central.com/showthread.php?2332472-A...

And here is the current roster of ads, though many are from external advertisers and hence less funny:

http://forums.somethingawful.com/adlist.php

(Note that "goons" is the moniker for forum members, and many ads target them specifically with "goon discounts" and so on.)

Marketers with an attitude and something of a free hand could take a leaf from them.

This is definitely interesting. I know I'd be more inclined to click something that looks like that ridiculous paint drawing. But this bit bothered me:

Results? 0.049% CTR vs. 0.137 CTR

I hope I'm not the only one confused by this, but the lack of a percent symbol on the 0.137 means he went from 0.049% to 13%, an improvement of almost 300x.

Is this a typo? Did he mean to say he increased his CTR to 0.137% or did he actually increase his CTR by 300x?

Edit: Why is this being downvoted? It's a legitimate question.

This has been known about for a long time; ugly sells.

People are more likely to trust something that looks amateur since it feels more like a recommendation than an advert.

Mr.Green, a well renowned CPA Marketer/Blogger, wrote a brilliant article about this. You can find it here - http://www.mrgreen.am/affiliate-marketing/the-ugly-truth/

I question _conversion_ though. Sure, the first ad had a lower CTR, but if you click on it, you almost certainly know what you're getting.

With the second it's very much "Haha, I wonder what this stupid ad goes to?" and then you just abandon it.

CTR means nothing without a conversion of some sort on the other side.

I wonder if the inspiration for this test was directly taken from Max Teitelbaum's interview @ mixergy?

Search for 'paint' and you'll find the relevant passage.

  "Andrew: Give me an example. I’d love to. 
  When we started media buying, we saw stock raising. 
  We saw what all the brands were doing. We needed
  something really flashy, something really clean.
  When I made something in Paint, as a joke and sent it over 
  to the guys, we just threw it up as a test, and it 
  quadrupled the conversion rate of anything else. 
  After that, I think we made every banner in Paint."

http://mixergy.com/teitelbaum-whatrunswhere-interview/

As joblessjunkie mentioned, novelty is a huge draw. I've seen ads on Facebook, Yahoo and MSM sites (probably delivered by ad networks) that feature eye-catching faces with some ridiculous or unusual element (long beards, tattoos, elongated teeth etc.), but are advertising something that's pretty mundane (loans, online education, etc.). They are much more noticeable than ads that use standard art or photographs.

Judging by the fact that the advertisers are still using weird face ads, the CTRs must be superior ... but the interesting questions would be around the type of customer or conversion rates.

I think it's critical to not lose track of buyer motives and end goals when testing ads.

If I were clicking on the first ad, I would do so with the intention of downloading Need for Speed. The second ad, I would click to answer the question, "Who made this crappy paint ad? It's pretty unique."

If you are paying per click, the first ad is going to perform better and will give more relevant traffic. If you are paying per impression and just care about traffic, the second wins.

I kinda take issue with his thesis:

"Every idea that you have is worth testing, no matter how crappy you think it is."

It took him all of 45 seconds to put that new test ad together. That is a very, very small time investment, thus making such a test worthwhile.

For instance, if one wanted to create some sort of live-action commercial with actors, lighting, equipment and CG, and test it as an ad, the investment of time and money would be so great that negative results would be disastrous.

The second ad is more "entertaining" in my opinion. Also "Ad Blinded-ness" might have caused the first ad to just go unnoticed among all the other ads. Being untypical helps on the web. But also, the point is that you cannot expect patterns to predict success always. The first ad was based on a pattern. Now, if we study the second ad for a pattern it might not work as well. Man, I love the web for this.

The novelness lies in the lack of polish. People who work in advertising like ads, and well-polished ads are more pleasing to their eyes.

To the rest of us, advertisements are used as something we don't want to see, but are willing to see, in order to see something we DO want to see. Many of us can pick out the disparity between the polished, targeted ads and the less-focused content it intersperses.

The MS Paint ad was much more readable, IMO. Too many adds try to cram too much into their restricted screen real estate, with the result that they're much less clear.

Factor in that most ads are competing with others alongside (and they all look the same); also as others have mentioned, we've adapted to filter out 'obvious' ads.

Yep, it's the same as when Google came out with their text ads. They were different, so people clicked them.

It might also have something to do with giving the impression that the game is going to be something novel, quirky or even bizarre, as opposed to another polished but bland EA title.

I think you're absolutely right. I would wager that if every ad on the internet was changed to a five-minute MS Paint scrawl, click-through rates would rapidly go back to where they were originally, if not drop even further.

80s bit-retro and rageface aesthetics.

Other pieces of universal marketing "truth", all of which are accurate/general enough to not be refuted:

1) Ethos (your perceived character) is the most important, with regards to pathos (emotion) and logos (logic)

2) People make judgments by comparison/anchoring.

3) People process information best from stories.

4) People are foremost interested in things that affect them.

5) Breaking patterns gets attention.

6) People look to other people's decisions when making decisions.

7) People will believe things more easily that fit their pre-existent mindset. The converse is also true.

8) People handle one idea at a time best.

9) People want more choices, but are happier with fewer.

10) People decide first, then rationalize - If people are stuck with something, they will like it more over time.

11) Experience is memory, the last part of the experience is weighted heavily.

As someone who has spent $100,000 on advertising over the last few years, at least $10,000 of which was on the Plenty of Fish platform, you're very wrong.

You only need about 1,000-10,000 impressions to get an idea of how a creative performs. Often less. As you get more and more used to each particular advertising platform, you also get a feel for how an ad is performing.

In my business, a difference of 0.02% CTR could mean the difference between earning 30% ROI and 50% ROI - the words "test everything!" mean everything to me and my results.

I agree with your point, however I would like to highlight your use of confidence intervals:

"If two statistics have non-overlapping confidence intervals, they are necessarily significantly different but if they have overlapping confidence intervals, it is not necessarily true that they are not significantly different."

http://www.cscu.cornell.edu/news/statnews/stnews73.pdf

In particular, it doesn't scale well at all with the number of choices (the familiar curse of dimensionality). You can reasonably test two variants, but if your design could plausibly vary along, say, 10 axes (not uncommon), you're going to have trouble collecting sufficient data to cover the whole 10-dimensional theoretical design space. So data-driven design can usually only be applied to a small part of the design space, typically testing a small number of alternatives.