Backblaze: Mounting Losses, Lawsuits, Sham Accounting, Insider Selling(morpheus-research.com) |
Backblaze: Mounting Losses, Lawsuits, Sham Accounting, Insider Selling(morpheus-research.com) |
> Instead, Backblaze’s new CFO, Marc Suidan, joined from Beachbody (NYSE: BODI), a multi-level marketing company
eek
I guess they could go door-to-door selling backup or maybe hold Backblaze parties…
TFA even specifies this:
> Instead they hired Marc Suidan who joined from Beachbody (NYSE: BODI) - a publicly traded health and fitness company where he was CFO between May 2022 and August 2024 when the company was operating as a “multi-level marketing” company.
[0] https://www.cnbc.com/2011/01/31/beachbody-grows-exponentiall...
[1] https://nypost.com/2024/10/01/business/beachbody-lays-off-th...
If you think he is a good CFO I think you drank the coolaid.
https://marencrowley.com/podcast/the-demise-of-the-beachbody...
And then promoting them also gives strong mlm vibes.
So I would bet more money on them being an mlm after reading your comment than before
I know I could use some open source stuff to get similar functionality, but I feel almost as nervous rolling my own backup as I would rolling my own crypto. Does Wasabi have some client solution that I'm missing?
https://hindenburgresearch.com/gratitude/
Funny, I wrote this without seeing what the full URL was (I’m on my phone) so the use of the word “gracious” preceded me knowing the path was /gratitude
honestly I participated in the IPO then sold when it was up a bit - I had faith in their fundamentals as a business but they did not seem mature as a public company.
Doesn’t that seem like insider trading? I guess he can claim he didn’t know so and so was going to quit.
That’s part of why they usually deputize a brokerage to periodically sell a fraction of their holdings at intervals.
It’s been a long while since I paid any attention to this but I do believe I saw at least a couple of instances of an insider triggering weakness in their stock just by increasing the rate of sale of their holdings.
But… are there securities laws in this era where policy decisions are timed to pump/dump the lowest?
If you're a less technical CTO, Wasabi seems to do a lot more traditional marketing on things like sports games: https://ibb.co/21qNkM9b (NBA, NHL, Premier League soccer). I'm not saying that makes them better, but a lot of people with purchasing power are going to be seeing Wasabi around.
Wasabi has also been pushing value-add services on top of their storage with Wasabi AiR (https://wasabi.com/cloud-object-storage/wasabi-air). It's basically an AI-powered database of your content. If you're running a place generating lots of content, having a system automatically tag and make searchable all those images and videos is a really nice value-add. Being able to search for a video clip containing a specific person in your library junk pile of video can be nicely useful for a lot of organizations.
It could just be that different people get their brand awareness from different places.
Small data, or low risk tolerance: Tarsnap
We replaced with Borg.
- Wasabi
- idrive
Who else?
Cloudflare, scaleway, hetzner, OVH, Oracle, minio and dozens of others all have s3 compatible object stores
Most of them expect AWS have at least inter cloud transfer free (or discount Azure and GCP) being part of bandwidth alliance(BWA).
AWS S3 pricing is designed to keep you in the ecosystem , not be the cheapest, so tend to be costliest not cheapest unless your workload is archival.
For almost all non archival workloads egress costs will be on par or much higher than storage costs and AWS[1] charges a lot for any bandwidth.
AWS also makes migration really expensive so if you started with them, you are likely stuck as you would have to pay up to 10-12 months of cost to migrate out if you are on anything but the standard tier (cheaper the tiers higher the retrieval costs)
[1] All tier one providers have 10-20x b/w costs compared to tier two , however Azure and GCP discount for inter cloud transfers AWS does not
It’s only Azure that isn’t on the S3 API train.
Although I appreciate that they included this, I can't help but feel it would have been more apparent if this statement had been included before the bullet points of the article. By the point I saw that statement, I had already formed an opinion on Backblaze. Although I am not saying Morpheus has done anything inherently dishonest, I do think the placement was intentional, and that it would have served the readership better if it had been the first thing that readers see.
> Initial Disclosure: After extensive research, we believe the evidence justifies a short position in shares of Backblaze (NASDAQ: BLZE). Morpheus Research holds short positions in BLZE, and Morpheus Research may profit from short positions held by others. This report represents our opinion, and we encourage all readers to do their own due diligence. Please see our full disclaimer at the bottom of the report.
Oh, and the finance employees refused to sign off on the books and a few are suing the company.
It sucks, but Backblaze is cooked.
But this is a hit piece. For any hit piece, I think it’s important for readers to know the motivations of the author, regardless of the validity of the claims.
(At least they disclose it at the bottom, but it should be in the title.)
this is activist short seller report/investigation, some of the points made:
- never been profitable
- execs selling aggressively
- loosing customers to Wasabi
- accused of cooking the books
- being sued by former execs for anti whistleblower/wrongful termination
- execs leaving
The only downside I found is that they limit free egress to 100% of your storage (so if you store 1tb, you get 1tb egress). In practice I don't think this will be an issue for my use case.
So I'm an example of a person who this morning had almost never heard of Wasabi, knows Backlaze well, and after an hour or two of research I have completely switched over.
Wasabi also treats every object as having a minimum 90-day storage period. So if you upload something and delete it, it's still considered "stored" for billing purposes for that 90-day minimum.
B2 also gives you 3x storage as free transfer compared to 1x for Wasabi.
I'm not saying you shouldn't go with Wasabi, but the reasoning you've articulated doesn't seem to track.
That's addressed in the article:
>Since 2021, Backblaze’s B2 Cloud Storage segment revenue growth has outpaced its legacy Computer Backup segment. In Q4 2024, the company announced that its B2 Cloud Storage segment generated $17.1 million in revenue, surpassing its Computer Backup revenue of $16.7 million.
Even if Wasabi isn't a straight replacement for Backblaze's entire business, it's a replacement for its biggest and fastest growing segment.
I ended up rolling my own solution with rclone and testing b2 and r3, going with b2 because of the price and speed.
Is there a Restic frontend for Windows that you'd be comfortable setting up a family member with?
That's where Wasabi is taking customers
Storage costs money. People love to dream up creative business models where your personal storage is subsidized by some other part of the business, but I think it’s just a matter of time before the business model changes. At the end of the day, there’s a massive gravitational pull that brings everything in line with market rates. These days, I think we have a pretty good idea of what market rates are for cloud storage. Anything less than market rates should be viewed with suspicion.
From another angle, in any long-term relationship with a vendor, you want the vendor to make money. We know that major cloud vendors do, or at least close enough, because they charge the same for small customers and big ones (or close enough). None of them offer unlimited data… or at least, not any more (most of them used to, but those parts of the business always get shut down).
This is the issue and it is a heuristic that people need to develop.
Your provider especially your backups provider needs to have interests that are aligned with your interests.
The non-B2 portion of Backblaze has a financial incentive to keep you from using their product and to use as little of their space as possible. This is a bad, misaligned set of incentives.
On the B2 side, all of our suspicions about the (well timed) "me too" IPO have been confirmed with each and every quarterly report showing increased debt load and the classic "losing money but making up for it on volume".
Now that money isn't free anymore it's just a matter of time.
The sad part is they don't use standard, commodity JBOD enclosures so there won't even be anything useful in the fire sale :(
Are you talking about their B2 product or their backup SaaS? The former has "fee that scales with usage", and the latter probably has enough normal users (ie. not data hoarders backing up 50 TB on the $99/year plan that they're not losing money overall.
When O365 launched, they were using spinning disk for exchange. The issue was that they stranded capacity because of the IOPS needs of exchange. So “free”, (low iops) SharePoint and OneDrive for business data utilized that “free” capacity.
Bandwidth costs can seem like a lot more, but when you purchase a 10 gig fibre, you have unlimited data, up until the full speed of the 10 gig fibre, 24/7. That number in TB can be calculated.
Clouds massively mark up services. Ones that underprice can have the opposite problem.
I have heard good things about the Backblaze service itself, and appreciate the hard drive reviews they put out.
The company is being plundered and run for the benefit of the exec team printing shares for themselves. Nobody should be buying shares expecting the price to rise.
It’s bits in a barn, and they’re not making rent. That’s worried me enough to re-evaluate my relationship with them, especially in light of this research.
Financial statements indicate they selling services at 50% margin, but all the extra sales/admin/R&D add up to a loss
They could cut all of that I guess, it's not like object storage need a lot of innovation and they've already stopped making their own storage servers and
The very quick high level explanation is that in storage you talk about "stretch factor". For every byte of file, how many bytes do you have to store to get the desired durability. If your approach to durability is you make 3 copies, that's a 3x stretch factor. Assuming you're smart, you'll have these spread across different servers, or at least different hard disks, so you'd be able to tolerate the loss of 2 servers.
With erasure encoding you apply a mathematical transformation to the incoming object and shard it up. Out of those shards you need to retrieve a certain number to be able to reproduce the original object. The number of shards you produce and how many you need to recreate the original are configurable. Let's say it shards to 12, and you need 9 to recreate. The amount of storage that takes up is the ratio 9:12, so that's a 1.3x. For every byte that comes in, you need to store just 1.3x bytes.
As before you'd scatter them across 12 shards and only needing any 9 means you can tolerate losing 3 hard disks (servers?) and still be able to retrieve the original object. That's better durability despite taking up 2.7x less storage.
The drawback is that to retrieve the object, you have to fetch shards from 9 different locations and apply the transformation to recreate the original object, which adds a small bit of latency, but it's largely negligible these days. The cost of extra servers for your retrieval layer is significantly less than a storage server, and you wouldn't need anywhere near the same number as you'd otherwise need for storage.
The underlying file system doesn't really have any appreciable impact under those circumstances. I'd argue ZFS is probably even worse, because you're spending more resources on overhead. You want something as fast and lightweight as possible. Your fixity checks will catch any degradation in shards, and recreating shards in the case of failure is pretty cheap.
Can't seems to find anything specific about Wasabi uses ZFS on Google. And Wasabi doesn't change you on egress. So I guess they are similar in terms of pricing.
Although B2 seems to be way more popular on HN. I rarely see Wasabi here.
And Warren Buffet would use many frontmen to slowly buy shares without triggering a buying frenzy. You can’t just buy or sell 3% of a company without causing drama.
I vaguely remembered they announced it, but didn't find it on a quick search, thought maybe I was wrong.
If your storage was fairly consistent, not tiny, and you intended to pull it regularly, Wasabi would come out on top. B2's egress change probably shifted the scales to favor B2 for those cases.
R2 was slower than B2, in my tests. I think it's slightly more expensive, too, but not much IIRC.
On one side, the cloud is someone else's computer they will always charge more an more for it because customers have not learned the economics of cloud hosting and how profitable it is on the other end.
On the other side, a NAS is your storage computer, in a simplified home appliance form. Connect it to back up to any cloud you wish.
If they're going to compress/decompress, ideally you'd want them to have that at the point of ingestion, then encrypted, and then store that on the target drive.
That way you can put very strong controls (audit, architecture, software etc) around your encryption and decryption, and be at reduced risk from someone getting access to storage servers.
For most regular people, I suggest Google Drive or OneDrive because you get the value add of the their ecosystem. With Google, Photos are good. With Microsoft, the Office+OneDrive subscription is a great value.
Can you elaborate on this? They mount in a standard rack and it looks like they have a backplane connected to a commodity motherboard. I'd think if you wanted it to be a JBOD you could replace the motherboard with an external SAS connector. Based on your nick I'm sure you've thought this through so I'm curious.
The only thing I see is that it looks like you have to slide the whole unit out to swap a drive, but that seems like an inconvenience rather than a show stopper.
I also think about all those cafés with free WiFi, where you buy a single coffee and take up a table for six hours while you browse Facebook and have your screenplay open in the background. I benefit, but the café doesn’t.
XFS or ext2 (or 3 or 4 wo journal), without LVM or mdraid.
There's no point to adding RAID at the hardware or OS level for object storage boxes when redundancy exists at the application level. A drive with too many errors will be marked "dead" and just spun down and ignored.
Metadata servers OTOH tend to be engineered to be much more reliable beasts.
Very interesting. Could you name a few, am curious. I would be happy if erasure codes are actually being used commercially.
What I find interesting is the interaction of compression and durability -- if you lose a few compressed bytes to reconstruction error, you lose a little more than a few. Seems right up rate-distortion alley.
The economies of scale mean that you really have to have something like erasure encoding in place to operate at large scale. The biggest single cost for cloud providers is the per-rack operational costs, so keeping the number of racks down is critical.
[1]https://d1.awsstatic.com/events/Summits/reinvent2022/STG203_...
[2]https://www.usenix.org/system/files/conference/atc12/atc12-f...
[3]https://cloud.google.com/storage/docs/availability-durabilit...
[4]https://www.backblaze.com/blog/reed-solomon/
[5]https://www.usenix.org/conference/hotstorage13/workshop-prog...
[6]https://research.facebook.com/publications/a-hitchhikers-gui... they even do some interesting things with erasure encoding and HDFS
[7] https://blogs.oracle.com/cloud-infrastructure/post/first-pri...
While it's not "commercial", but CERN uses it and many other institutions.
https://indico.cern.ch/event/941278/contributions/4104604/at... --- 50PB
...
ah, okay, finally an AWS S3 presentation that mentions EC :)
https://d1.awsstatic.com/events/Summits/reinvent2022/STG203_...
"Uh, what?? Air isn't an MLM, it's just the thing we breath. We all need it to live."
"Huh, I guess air is an MLM after all. Wild."
Vector Marketing, for example — the company that sells Cutco knives door-to-door — might be incredibly scummy, sure. Their entire business model is to
1. talk college students into thinking they can make a continuous monthly profit by selling $800 knife sets (when really the profit is one-time at best, by tapping into each college student's family and friends — who are only sympathetic enough to buy the knives [if they even are], because it's their family/friend asking);
2. forcing those college students to "buy into" the company, purchasing a knife-set of their own to use in sales demos (which they can't return if they quit);
3. and also forcing those college students to recruit other college students on campus.
But, because every transaction is ultimately just lining Vector Marketing's pockets directly — without any revenue structure involving making more money as you recruit others whose sales "become your sales" — it's not Multi Level Marketing.
It's just sleazy.
(And yes, this is just me taking this opportunity to rant about Vector Marketing. I have half a Cutco knife set laying around, from when my college roommate attended a "job offer" and got essentially bullied into buying a set before they could leave. Every time I use it, I think of how tarnished the Cutco brand is by these awful tactics of Vector Marketing's practices. Which is a shame, because they're decent knives. But obviously not knives I'd ever recommend anyone buy, because I don't want to support a company that thinks it's a good idea to work with a company like Vector Marketing.)
How do you test your backups? You do test your backups, right? Your other comment mentions using dropbox, but that's hardly a real backup solution, and you could easily run into a situation where you need to retrieve files beyond the 30 day window.
And it’s such a good thing to remind people of. You don’t manage a backup service. Nobody cares about backups.
People care about restores.
I would personally be fine running tests on data that is recently backed up and only testing the data in glacier once or twice. Think about why you test backups in the first place—the main errors you’re trying to catch are problems like misconfiguration, backups not getting scheduled or not running, or not having access to your encryption keys. You can put your most recent backup in “infrequent access” and let older objects age out to glacier with lifecycle rules.
Glacier used to have really expensive retrieval costs. That’s now called “glacier deep archive” and as far as I know, major use cases are things like corporate recordkeeping / compliance (e.g. Sarbanes-Oxley). The costumers for deep archive should be sophisticated customers.
No - no exceptions.
We must, as sophisticated 21st century actors, insist that providers (especially providers of critical services) have financial interests that align with our own interests.
I don't care how big the parent is or how much money they have to burn - if it makes financial sense to keep you from storing your backups you need to go elsewhere.
Non-nerds are going to screw up backups without good UI. If you want to pay by the drink, like I said, use the hyperscalers.
If it was aligned with economies of scale, customers would get more storage for the same price every year.
We don't hear about cloud prices coming down that often.
I know nobody who use them, so it all balances itself.
Gosh, if only there were a cloud storage provider that gave you an empty UNIX filesystem you could do anything you wanted with ... maybe something built totally on open standards that could handle hardlinks and symlinks and such ?
If only ...
It’s not the same service as backblaze’s client but it does everything I need, with dedupe
$99/year with no limits on data, as far as I can tell. The context here is someone talking about personal backups, not Backblaze’s broader offerings. We know the market rates for cloud storage, more or less, since they’ve converged somewhat, across the industry. This makes it easy to calculate what kind of usage patterns would lead someone to save money at Backblaze.
Cloud services often have a free tier or cheap tier to attract new customers. IMO, this is not it. This is a product. These people are not turning around and signing contracts with Backblaze at work. At least, not many.
So it could be that people are just using it below its limits, or it could be that it’s subsidized by other parts of the business. Overall P&L is irrelevant—you want to be able to explain this product as a profitable product, as some viable sales channel, or as marketing.
My recommendation is to buy storage products where the company is making money off you, or nearly.
The limit is as much data as you have plugged in. Backblaze only backs up your currently-available data, with a 30-day grace for external disks[1]. IMO, if you have a computer with a 1TB drive, you'll probably use <1TB at all times, accounting for you not using the entire 1TB locally and your changed data probably fitting in the excess. (e.g.: 800GB used space and you change <200GB over 365 days)
1: https://www.backblaze.com/computer-backup/docs/external-hard... . Essentially, if you unplug an external drive for >30 days, that drive's backup will be deleted.
So do the calculations? As I mentioned in another comment, their unit economics are sound, with them making more than 50% gross margin.
>So it could be that people are just using it below its limits
That's how many businesses work. Many low end gym chains operate on the assumption that most users won't use their services. There's no way that planet fitness can equip and operate a gym for $10/month per member, if everyone one of them went regularly. That doesn't mean it's a bad idea to get a planet fitness membership (assuming you'd actually go), or that they're at risk of going under.
Is this some kind of personal challenge? It’s arithmetic for chrissakes, not calculus. S3 IA $0.0125/Gb-mo, ingress basically free, $99/yr÷(12mo/yr×$0.0125/Gb-mo)=660 Gb. Reasonable to ignore egress for backups—you may want to account for it, but it’s reasonable to ignore it. Do your own calculations if you have different assumptions.
That’s not hard. The numbers don’t have to be exact.
Maybe I just think it’s easy because I’ve run numbers like these for a living. Given X different storage configurations, under which conditions is configuration 1 cheaper than configurations 2 and 3?
> That's how many businesses work. Many low end gym chains operate on the assumption that most users won't use their services. There's no way that planet fitness can equip and operate a gym for $10/month per member, if everyone one of them went regularly. That doesn't mean it's a bad idea to get a planet fitness membership (assuming you'd actually go), or that they're at risk of going under.
The gym closes I can buy a different gym membership. My cloud storage service closes, it may be difficult to move my data to a different provider, depending on how much data and what kind of rate limits. We’ve seen this before. People with data stuck in an “unlimited” storage service that’s shutting down, trying to transfer it but getting their egress throttled.
The economics of gyms are different because the cost is not only the membership, but also the time it takes you to get to the gym. Cloud services are a much more efficient market, which means the margins are much lower.
There's no way that planet fitness can equip and operate a gym for $10/month per member, if one of them simultaneously used all machines at all locations.
"The developers at Klara Inc. have been outstanding in helping Wasabi resolve ZFS issues and improve performance. Their understanding of ZFS internals is unmatched anywhere in the industry" - Jeff Flowers, CTO, Wasabi Technologies
You could also search the OpenZFS repository for commits with the word Wasabi in them.
Wasabi only allows as much egress as the amount of data you're storing and I don't think you can even pay for more: https://wasabi.com/pricing/faq#free-egress-policy.
There's a number of CDNS that participate in zero egress.
(Unless something’s changed since the last time I checked)
EDIT: OP is correct for CDN but if you use R2 even as a transparent copy from another S3 like provider it is allowed [1]
This includes AWS's margin, which could be quite handsome. Bandwidth costs for the top 3 hyperscalers have converged to around 9 cents per GB, but nobody seriously thinks that's anywhere close to to the actual cost of bandwidth, or that hetzner/ovh is going to go under because they're offering 1TB ($90) of bandwdith for their $5/month VPS.
Even taking the 660GB figure at face value, that's plausibly within the range of what I'd expect the average person to have backed up, especially when you consider that the pricing is per machine, and the standard desktop/laptop comes with around 500 GB of storage.
You've also failed to address my point about their gross margins. The reliability of their books have been questioned, but even the short report didn't accuse them of cooking that metric.
>The gym closes I can buy a different gym membership. My cloud storage service closes, it may be difficult to move my data to a different provider, depending on how much data and what kind of rate limits. We’ve seen this before. People with data stuck in an “unlimited” storage service that’s shutting down, trying to transfer it but getting their egress throttled.
It's a backup service. It's not even trying to compare itself with the likes of dropbox or onedrive. You should have at the very least a second copy locally.
It could not be.
Storage costs are more or less race to the bottom. Storage business model is that it’s cheap and commodified, but sticky. Your data is stuck in S3 because it’s a pain in the ass to migrate. The margins come from non-commodified parts of the cloud (higher-level SaaS products) and I suspect egress costs (I don’t have the details).
There’s three parts to my understanding here. One part is my understanding of the business model, which explains why storage is a race to the bottom, why it’s not subsidized (or not subsidized much), why from a strategic perspective, the margins are getting eliminated. The second part to my understanding is the technical aspect—the calculations for how much it costs to stick hard drives in a data center and run storage systems on top with the desired accessibility and reliability, and I have some fairly deep expertise in parts of this. The last part is my personal experience working for cloud providers.
I can elaborate on the first two parts but for obvious reasons some of my knowledge is proprietary and I can’t share it.
> Bandwidth costs for the top 3 hyperscalers have converged to around 9 cents per GB, but nobody seriously thinks that's anywhere close to to the actual cost of bandwidth, or that hetzner/ovh is going to go under because they're offering 1TB ($90) of bandwdith for their $5/month VPS.
Bandwidth is not commodified like storage. I know that storage cost is close to operational cost because I’ve done extensive calculations and comparisons—and some projects are cheap enough that you have to revisit your theories about how they work on a technical level.
I don’t know why bandwidth is expensive. I’m not an expert on bandwidth or CPU or 90% of the other stuff. I only happen to have expertise on storage.
There are services you can buy where egress is the major cost and egress is significantly cheaper than egress from AWS, GCP, or Azure. Either egress is a complicated product or the margins are high. Probably some combination. Storage is less complicated and the margins are lower.
> You've also failed to address my point about their gross margins. The reliability of their books have been questioned, but even the short report didn't accuse them of cooking that metric.
I guess I don’t understand what point you’re making, or how that point about gross margins makes sense. It doesn’t make sense to me so I’m not sure what I’m supposed to respond to.