Someone alerted this to me afterwards. Unfortunately I only recognize him by a different username LOL

He was so understated that I missed it lol

The Rails view layer used to be quite nice at its peak. Way better than many of the alternatives. That you could execute actual code inside a view was just magical.

That is absolutely absurd that you have built something many have tried and failed to do with millions of dollars of venture capital behind them, all on your own.

Genuine kudos to you, you should be an inspiration to any indie hacker.

I've written an overview about the differences between Nylas and EmailEngine here: https://docs.emailengine.app/emailengine-vs-nylas/

TL;DR Nylas is hosted service that does way more than just emails. Also, it syncs all emails on monitored accounts to their DB, so when you make an API request, it hits their DB. EmailEngine is self-hosted and runs directly against the mail servers, so when you run an API request, the request ends up in the mail server. Both approaches have their pros and cons. Other than that, there aren't any other good solutions than building everything yourself by using any IMAP library.

Sorry, this is bogus.

"If I had asked people what they wanted, they would have said faster horses." or whatever the quote is.

There are tons of businesses selling products that are ideas brought to life from scratching their own itch or simply a desire to make something and put it out there.

Personally, I am one of those people who started a business based on an idea with no validation before launching it.

I built it in its entirety, then went to places and people who I expected would want it and low-and-behold, I am making a living doing it.

Most EmailEngine customers are smaller SaaS services like niche CRM providers etc. EmailEngine can also work with Gmail and MS365, as they support IMAP in addition to their proprietary APIs. So you get quite a large coverage by using EmailEngine out of the box (all smaller providers + most Gmail/MS365 accounts). Additionally, in some cases going with IMAP is the only way - for Gmail email API access, you need a very expensive security assessment that you might also fail for various reasons, like when your use case is not allowed by Google's terms. In comparison, there are no conditions for using app-password-based IMAP for Gmail. EmailEngine does not work with email accounts where their organization has disabled IMAP/SMTP access entirely (common with on-prem Exchange servers, a lot of MS365 accounts, etc.)

I think you're taking this way too seriously, it was just a blogger having a little lighthearted jab

> It costs nothing to actually put down a reason for being upset, versus having totally generic downputs

Actually, it does cost something- time and effort- but thanks to the latest AI, I can cut some of that out right now:

As an Elixir enthusiast, I'd like to point out some technical criticisms of JavaScript and its ecosystem, particularly in comparison to Elixir's approach to dependency management and community.

A) Dependency bloat: The JavaScript ecosystem, especially when working with Node.js, often suffers from excessive dependencies. Even when you require just a few dependencies for your project, you may end up pulling in a large number of indirect dependencies. This can lead to increased complexity, slower build times, a larger attack surface for potential security vulnerabilities, and an ever-growing workload spent applying security fixes that have nothing to directly do with the purpose of the web app.

In contrast, Elixir's dependency management system, powered by Mix and Hex, encourages a more conservative approach. Elixir libraries are often designed with minimal dependencies, and the community values self-contained, focused libraries.

B) Security concerns: The extensive use of third-party packages in JavaScript projects can expose your application to security risks. When you pull in a large number of packages, you become responsible for ensuring that all of those dependencies are up-to-date and secure.

Elixir's ecosystem tends to have fewer dependencies, reducing the potential attack surface. Additionally, Elixir's focus on immutability and the actor model can help minimize the impact of security vulnerabilities.

C) Lack of native concurrency and parallelism: JavaScript is single-threaded by design, which can limit its ability to take full advantage of multi-core systems. While Node.js introduced asynchronous I/O to help mitigate this issue, it can still be challenging to build highly concurrent applications in JavaScript.

Elixir, built on the Erlang VM (BEAM), provides excellent support for concurrency and parallelism through lightweight processes, message-passing, and the actor model. This allows Elixir applications to scale across multiple cores and handle a large number of simultaneous connections more efficiently than most JavaScript applications.

D) Callback hell and async/await complexity: JavaScript has traditionally relied on callbacks for asynchronous programming, which can lead to a phenomenon known as "callback hell" when dealing with nested callbacks. While the introduction of async/await has improved this situation, it can still be cumbersome and confusing for developers.

Elixir offers a cleaner approach to concurrency with its process-based model and message-passing mechanism, enabling easier-to-read and maintainable code.

E) Mutable data- One of the key differences between Elixir and JavaScript is the way they handle data. Elixir enforces immutability, meaning that once a data structure is created, it cannot be modified. Instead, operations on data structures in Elixir return new versions of the data, leaving the original untouched. This approach offers several advantages compared to JavaScript's mutable data structures:

  1) Predictability and easier reasoning: Immutability in Elixir makes it easier to reason about and understand your code. Since data structures cannot be changed once created, you don't have to worry about unintended side effects or data being altered in unexpected ways. This results in more predictable and maintainable code.

  2) Concurrency safety: Elixir's immutable data structures simplify concurrent programming by eliminating the need for locks, mutexes, or other synchronization primitives. This is because multiple processes can safely read and share the same data without the risk of data corruption due to concurrent modifications. In JavaScript, managing shared mutable data in concurrent scenarios can be error-prone and challenging.

  3) Functional programming: Immutability is a core principle of functional programming, and Elixir is heavily influenced by this paradigm. Functional programming promotes the use of pure functions that do not produce side effects, which can make code more modular, reusable, and easier to test. While JavaScript supports functional programming concepts, its mutable data structures can make it more challenging to adhere to functional programming principles.

  4) Reduced cognitive load: Immutable data in Elixir means that developers don't have to keep as much track of changing state as they read or write code. This can make it easier to understand the flow of data and logic in your application, leading to a more pleasant and efficient development experience and ultimately, fewer bugs generated per LOC.

F) Community values: The JavaScript community is known for its rapid pace of change and the constant introduction of new libraries, frameworks, and tools. While this can drive innovation, it can also lead to fragmentation and inconsistency across projects.

The Elixir community tends to be more focused on stability, long-term support, and collaboration. This can result in a more consistent and cohesive ecosystem, where tools and libraries are more likely to work together seamlessly.

In conclusion, while JavaScript has its strengths and has undoubtedly played a vital role in the growth of web development, its dependency management and ecosystem can be seen as less than ideal from an Elixir enthusiast's perspective. Elixir's dependency management system, focus on concurrency, and community values offer a more robust, sane and secure alternative.

(How's that for "actually putting down a reason" as to why Javascript is terrible? Hey, you asked!)

This is a callous take that refuses to pause for even the briefest millisecond of reflection. Personal attacks like this & shaming me may make you feel good & smart but as far as I can tell there's still a real topic here & it still deserves some discussion.

I don't go out of my way to start this discussion every time. But sometimes I do. Because it's super notable to me how asymmetric it is. If there was more evidence of programmers dropping little nasty grams about any old topic, I could see it as cultural. That's why I tried to survey the room. It's remarkable to how persistently nasty people are to the most popular language on the planet. It feels decidedly unhacker like to bully me into silence over this search for meaning & inquiry.

I'd like an example of programmers persistently being mean & denigrating to other programmer spaces or languages. That was the context of the rest of my line of questioning, and implied by this question too. The popular/mainstream perception of programmers is unimportant & distracting, disinformation that misleads the discussion; why do some programmers enjoy dropping little vacuous nasty gram jabs at JS, and is this pattern repeated broadly against any other targets?

Are we concluding that JS is the only bad programming thing, thus no one else gets mocked? Maybe JS is a magnet given how popular it is? Or do we think the casual barbs really do go every way & I'm just missing it?

The price is fine, I'm more so talking about not having a pricing section on the main website and instead being on another website domain.

> If it was good enough reliability for GitHub and Shopify to use for years, it was good enough for Sidekiq OSS too.

Great point, and thanks for chiming in. I wonder if containerization has made this more painful (due to cgroups and OOMs). The comments here are basically some people saying it's never been a problem for them and some people saying they encounter it a lot (in containerized environments) and have had to add mitigations.

Either way, my observation is a lot of people not paying for Sidekiq Pro should. I hope you can agree with that.

Idempotence doesn't solve this problem. The jobs are all idempotent. The problem is that jobs will never be retried if a crash occurs.

This doesn't happen at a high rate, but it happens more than zero times per week for us. We pay for Sidekiq Pro and have superfetch enabled so we are protected. If we didn't do so we'd need to create some additional infra to detect jobs that were never properly run and re-run them.

Jobs may crash due to VM issues or OOM problems. The more common cause of "orphans" is when the VM restarts and jobs can't finish during the shutdown period.

We process around 50M sidekiq jobs a day across a few hundred workers on a heavily autoscaled infrastructure.

Over the past week there were 2 jobs that would have been lost if not for superfetch.

It's not a ton, but it's not zero. And when it comes to data durability the difference between zero and not zero is usually all that matters.

Edit for additional color: One of the most common crashes we'll see is OutOfMemory. We run in a containerized environment and if a rogue job uses too much memory (or a deploy drastically changes our memory footprint) the container will be killed. In that scenario, the job is not placed back into the queue. SuperFetch is able to recover them, albeit with really lose guarantees around "when".

it’s not uncommon to lose jobs in sidekiq if you heavily rely on it and have a lot of jobs running. If using the free version for mission critical jobs, I usually run that task as a cron job to ensure that it will re-try if the job is lost.

I have in the past monitored how many jobs were lost and, although a small percentage, it was still recurring thing.

In containerized environments it may happen more often due to OOM kills or if you leverage autoscalers and have long running sidekiq jobs that have a runtime that exceeds your configured grace period for shutting down a container during a downscale and the process is eventually terminated without prejudice.

OOM kills are particularly pernicious as they can get into a vicious cycle of retry-killed-retry loops. The individual job causing the OOM isn't that important (we will identify it, log it and noop it), it's the blast radius effect on other sidekiq threads (we use up to 20 threads on some of our workers), so you want to be able to recover and re-run any jobs that are innocent victims of a misbehaving job.

If you want to focus on durability then I think your complaint makes even less sense. Somehow I doubt S3 is primarily backed by Redis.

I think it's fair to assume that something backed by Redis is not durable by default because that's not what Redis is known for, whereas the other options you listed are known for their resiliency and durability. I wouldn't view Sidekiq as a similar product to RabbitMQ and SQS.

Also, Sidekiq Pro uses more advanced Redis features to enable super_fetch lending to the assumption that by default Redis is not durable: https://www.bigbinary.com/blog/increase-reliability-of-backg....

To me 'crash' means any unexpected termination, whether it's caused by an uncaught exception, OOM, or hardware/network issues.

I guess you can say that hardware issues on your host aren't under your control, but it's under your control to find a host that doesn't have these issues. And not even a full-on ACID database is going to be 100% reliable if you yank the power cord at the wrong moment.

For sure, I agree with you.

I would say that queued jobs being lost is different from an in-flight transaction being auto-rolled-back, but it's not a super important distinction. Like others have said, I think Sidekiq really nailed the free vs premium features and its success is evidence of that.