Slightly off-topic but Vercel's docs are really awesome these days. Good job Leerob and co. it has all come a long way in only a few short years. Not much left for me to bitch about ;)
Also feel free to join our slack for more feedback :D
Our goal is to provide some out-of-the-box alerting tuned to Vercel deployments as we learn more about how devs are using Axiom & Vercel!
Couldn't be simpler. Team's great too. Highly recommend
We use Axiom at Prisma for various things and love it. Happy to answer questions if anybody is interested. Either here or on email.
It's a fair point, though...at some point, and with enough TBs of traffic, it could get expensive to provide the log drain support. One thing we've been working on (and more on this in the coming weeks), is putting together a benchmark of log-drain mechanisms to try and work out what would be the most efficient combo of compression/cpu/mem to use for log-drain providers.
Our hope is we can work on making it efficient enough that we can encourage more services to provide log-drains (and therefore allow us to make great experiences around those logs!).
But also, you can sample some of the data within your own VPCs (see Honeycomb's Refinery or Lightstep's Satellites)
I'm not being critical, I'm genuinely interested
Existing solutions, and those architected in a more traditional way than Axiom, would require highly-coordinated nodes running on expensive VMs that would be bound by cpu, memory, and storage depending on your use case:
- Want to store TBs of data for months/years and query any piece of that at any time? Prepare to have expensive SSDs or wait for 'archived' data to swap in from cheaper storage.
- Want to run a query that combines N datasets, calculates aggregations, over TBs of data, and then compare that against data with the time shifted back a week, month, or year? Great, fire up some heavy VMs with enough cpu, memory, and bandwidth to compute all that.
- What if you use case varying greatly in each dimension (how much ingest you'd use, how much storage you'd need, and how heavy your queries would be) across a day, week, or month? You'd have to find a way to adapt to changing requirements or just scale up for the worst case x2 and just pay the $$$.
With Axiom, we had three key goals:
- Hyper-efficient, co-ordination free, schema-less, and index-free ingest (~1.4TB/day on a $5/mo container)
- Cheapest storage for hot, cold, archived and warehoused data: all data is stored in object storage, highly compressed & ready to query. 10 seconds or 10 years old data is the same, already as cheap as possible.
- Serverless querying that can expand from 0 to as-much-as-AWS-will-allow depending on what your query needs + how many of your team are querying at once.
The above was achieved through a lot of trial-and-error, tweaking, testing, and head-scratching, but we're really proud of what we've built. We can run super-fast ad-hoc queries, we've eliminated the need to think about retention, we have live streaming, and we have an incredible query language inspired by Microsoft's Kusto (Splunk users will be familiar!).
We still have a lot I'd like to see done, but I think we really do have something unique :)
And I'm glad you went to the lengths you did in your explanation!
Those specs are _insane_. What was your stack other than S3?
Do you have plans on PaaS/SaaS-ing your tsdb?
I've kind of taken it on as my life's work to study the software development process when it comes to "Product vs Platform" solutions. It's something I'm passionate about, and I'm forming strong opinions, loosely held for now.
So often in my (still short) career I've encountered so many "we couldn't find the right fit and so we built our own"-isms, and with very, very few have I ever felt like it was necessary.
Here is the rough draft of my opinion on internal solutions vs 3rd party/FOSS solutions. It's in the heat of building a series of internal solutions that aren't necessary, given the constraints and resources available.
https://mikercampbell.bearblog.dev/build-tech-or-product/
I'm currently working on a blog post that explains how (this time with more tech than just opinion) 95% of tech problems could easily be solved by 5% of the technology out there.
With the traffic and demands you have, you are of the 5% of tech problems where it isn't improbable that you couldn't find a tech to meet your needs.
I just need more experience in the field of tooling before I feel confident putting my foot down anywhere and drawing the line. So I'd love to talk more if you're open to it!