Results from Candy Japan box design A/B test(candyjapan.com) |
Results from Candy Japan box design A/B test(candyjapan.com) |
One of our all time most successful tests was related to this story and happened almost by accident.
The client had just rolled out new packaging and wanted help validating the change statistically, very similar to this situation.
Their metric was measuring upsells to a premium product after a purchase, and like Candy Japan, found statistically insignificant results.
But the new packaging was beautiful and we wanted to test it on their landing page.
We tried just replacing the old product image on landing page. Meh results, no real bump.
We tried increasing the size of the image + circling with "Improved" (no joke). 10% bump. Interesting.
We hired a professional model to hold the product and smile. 15% bump. !?
We hired a group of professional models to hold the product and smile. 40% bump in sales. !@%K$!?
We re-hired the same group of people to hold product, look SUPER excited (jumping up and down like their team just won the superbowl). 50%+ bump in sales.
No joke.
If Candy Japan is reading and you'd like some help recreating I would not be shocked to see something similarly group oriented, visceral, and beautiful bump yeh.
>10% bump.
>15% bump.
>40% bump in sales.
>50%+ bump in sales.
Was that cumulative (i.e., 1.1 * 1.15 * 1.4 * 1.5 ≈ 2.7 total) or relative to the base figure?
How many sales would Amazon have today if they went back to their 2000s website look?
The decorated box might not even have the best drawing, but indicates that the salespeople went the extra mile
In my mind's eye, that'd remain classy enough not to be creepy/cringy/embarrassing (as https://news.ycombinator.com/item?id=11641715 complains), but would also have sort of a personal touch to it.
Having a couple rubber stamps made and stamping each box during packaging wouldn't probably cost an awful lot...
After all, his main edge over the local japanese stores is authenticity. The fact that it comes "directly" from Japan.
In other words, the old box, while plain, adds a certain intimacy. It doesn't look like a huge multinational candy company sent it. The new box, though, looks basically like any other piece of Japanese media.
The Japanese illustrated box is innocent and kind of cute (certainly not "rapey" as another commenter put it) and in a Western country, stands out from the things around it. Perhaps if you're in Japan, surrounded by Japanese media it might seem generic but I don't think that's a problem elsewhere.
* It looks childish. If you are familiar with the Japanese style of advertising etc., probably not an issue, but as a box to be sent anywhere and to be received in a variety of settings (workplaces etc.) it seems like a poor choice.
* Anime may have poor connotations (unfair IMO) as pointed out elsewhere in the thread, which will reflect on the receiver.
* It feels corporate / like advertising. It may seem silly, but I'm more likely to continue subscribing to a small company if I'm on the fence about canceling.
* It feels like a waste of money. The current packaging is utilitarian, which signals that money is being spent only where it needs to be. The new packaging feels like an unnecessary indulgence, which would make me think I'm getting less value out of the subscription.
- The anime picture is embarrassing to receive in the mail / at work.
- The picture itself is kind of creepy, someone walking past my desk once even described it as 'rapey'.
- I can reuse the box.
- I am not 6.
Have you tried to run introduction offers? 50% off for first shipment for example? If I were you I would try to put the countdown for the next shipment on top of your site. Thats a really powerful psychological factor.
Your CTA is also a little unclear, why do I need to first get an email, then a link? The buy process should be straight forward, not through my email. And if you have a really good reason for doing it this way, you should atleast tell me why I need to leave my email.
I really like the cartoon that explains the process. Very good.
Regarding packaging design, we did a huge remake in 2013. We increased our prices with 30%, and redesigned packaging completely. Doubled our LTV, insane gains. Just keep testing, but try to change provider for carton. We pay $0,1 USD for bigger cartons then yours, in one of the most expensive countries in the world. (With plastic window for address field) We chose to put our three USP on the back of the shipment box, to remind your customers every time they receive a shipment why they subscribed in the first place.
Keep testing!
* What does LTV mean?
* What does USP mean?
USP = Unique Selling Point = Why your subscription gives the subscriber something they can't get anywhere else
Why not poll your subscribers or (assuming you can get the costs to work out) have an option for anime art. Personally I wouldn't want the anime style box, something custom and simpler would be fine.. but thats my personal opinion.
I suddenly get p-values now. Why has no one ever explained it like this before?
Which package design would you rather receive?
The white one with illustration 79%
The plain dark one 21%
Total votes: 52
The white box would be better advertising for you in my case :)
If I went to a friends house and saw this, I would never remember the name. If you could combine "making the purchaser look cool" with timing-relevancy, I think your packaging improvement would pay dividends.
Put another way, I believe your A/B ratio test is actually X*Y/Z where X/Z and Y/Z are both < 1. Synergy.
Terrible idea, (possibly) useful for analogy: Instead of optimizing the unboxing experience alone, if you also included cards with QR codes, where every QR code immediately triggers a free candy bar (with user data entry on a mobile app). It should also provide social value to the host in realtime; maybe they also receive an immediate notification that their guest scanned their card plus some other bonus I can't think of.
The point is to get closer to the conversion/churn decision, or provide more value to your consumer.
I used to receive Candy Japan in the "plain" box to Australia, and the customs sticker declaring the value & contents took up most of the space on the box.
https://www.quora.com/What-statistical-test-should-be-used-f...
(Of course, even better would be a Bayesian decision analysis which takes into account the full posterior distribution, costs, and benefits, allowing one to decide whether the evidence is enough to justify the more expensive box or whether one should extend the experiment further to collect more data.)
* quick, if conversions are 0.25%, what does a 0.1SD mean difference between arms of an A/B test mean? 'Uh...' Exactly.
All of this is "I think" and "IMO" of course.
was the old box that blue plain box? It's not intimate to me. Compared to the new one, it looks kind of homely
This person may have some deeper issues than just the innocent picture on this box.
Someone who admires Japanese culture will likely not have the same reaction as someone with little exposure to Japanese culture.
The picture is not even on the same planet as 'rapey' Whoever that was either said it to get a rise or they have some distinct issues.
One solution, perhaps, is avail both packaging options to buyers.
The typeface and font, I think, could be improved. It kind of looks like an afterthought. I don't see "branding". It's not Lobster or Comic Sans... but...
One problem is that even if the person receiving the box is exposed to Japanese culture, their coworkers probably aren't.
I think you're projecting your own issues quite considerably.
Maybe I should start calling this the plain vs. rapey test.
It's a box of candy. It's not a metaphor for sex, it literally contains candy.
Maybe I'm the weird one here?
[1]Just look into Rorita fasshon, etc.
I'm not sure what you mean. If all you have is binary data, there's not much you can do but a chi-squared test or binomial regression. As I understand it, a chi-squared is considered nonparametric: https://en.wikipedia.org/wiki/Chi-squared_test https://en.wikipedia.org/wiki/Fisher's_exact_test
*The age is hard to tell, it's like trying to divine Buggs Bunny's age from his cartoons. But the age is irrelevant as it's two people with a box of sweets.
You're still right, though.
You mean "kawaii," right?
You might not like that the world works that way, but as a business it's wise to not deny reality.
Wait, what? In fashion, sure, by definition - but when I'm choosing between Persil and Tide, or Coke and Pepsi, or Pizza Hut and Domino's, I'm primarily looking at how good it is at whatever I want it to do. Which one cleans better, which one tastes better?
There's a certain point where many people start buying things more as social signalling than as actual useful things, but I'd disagree that the vast majority of brands could be fashion items in this way. The rest use their brands to evoke trustworthiness - "I know that this will do what I want".
That said, that does not explain people reading the kinds of graphic novels and magazines they do in plain sight in public on trains in parks, etc.
Trains and parks, by contrast, are generally considered an appropriate context to consume these kinds of materials. That being said, book covers are also far more commonly used for exactly the same reasons.
_That_ person is the problem in the office, not the person who bought sweets online and probably shared them with some of the office.
If anything, the packaging helped reveal an oddball at the office.