Ruben here, software engineer, long-time lurker of Hacker News and founder of Windmill. Windmill is a fully open-source self-hostable platform and runtime to build complex workflows, internal apps and integrations using any scripts in Python or Typescript-deno. I am back after having been revealed a bit too soon on HN and miraculously getting into YC (https://news.ycombinator.com/item?id=31272793).

To build internal apps for ops, integrations between services that cannot talk to each other directly, or to run background jobs that run your business logic and analytics, the two main options today are no-code solutions and old-fashioned, roll-your-own scripting. Both have problems, and our goal with Windmill is to find a new sweet spot between the two. No-code solutions are productive if your problem matches the tool exactly - but it not, they are rigid, hard to extend and quickly become tech debt, annihilating their initial time advantage. Indeed, no-code is just code but made by an opinionated someone else and hidden as a blackbox with an UI.

The alternative is to do it the old-fashioned way, writing everything from scratch, both backend and frontend, perhaps deploying it on the latest flavor of serverless, and pray to never have to touch it again because that took way too much time and it has now became a burden that the ops and business team might poke you about regularly.

Furthermore, the landscape of SaaS is specialized tools for everything—alerting, data analytics, administration panels, support management, integration between services—when it feels like a few scripts would have been as good or even better and spared you the need of depending on one yet another tool. This could be even further facilitated if there was a way to import the right bunch of scripts from a fellow community of engineers, tweak it and deploy it like you can do in communities where automation can be shared as simple JSON files, for instance in the node-red or home assistant community.. That’s the idea of Windmill: to bring back the power of scripting in an easy way.

With Windmill, you write normal scripts, or reuse ones made by others, and we make them production-grade and composable. You shouldn’t have to worry about things like http requests or scheduling jobs. We abstract much of that away, making your scripts be both more focused and more composable. You end up doing things the right way but much quicker.

We reduce the complexity of workflows, integrations and internal apps by uniting them all under one banner. At the heart, they mostly have the same needs: workflows with a UI or a schedule. One tool that does it all out-of-the-box offers greater consistency and allows you to grow the complexity of your toolset at your own pace.

I have an academic background in compilers and industry experience in distributed systems. My compiler work made me wary of solving every problem with a domain-specific-language (DSL) or complex frameworks. We can just do more with the well-crafted existing languages like Python or Typescript. Rolling up your own DSL is nice in theory, you can make it very ergonomic and focused on the task at hand, but then you start adding features and either reinvent existing – albeit worse – programming language or decide to stop there. In the very large majority of cases, a well crafted library is vastly superior to any DSL. By being able to use any library of Python and Typescript, we stand on the shoulders of giants.

I have also observed that the best distributed systems are often the most simple as they are more predictable and have invariants that are easier to reason with and scale horizontally. This is why for Windmill, we rely solely on Postgres + our native workers + our http REST api layer. Later on, we plan to build adapters to host the workers on AWS lambda or Cloudflare workers, and the queue on Kafka if your needs are exceptionally high.

At the heart of what we have built is a queue implemented in Postgres and workers implemented in Rust that create a sandbox (using nsjail), fetch dependencies, and execute scripts. Every script can be triggered through its name with an HTTP POST by passing a JSON payload in which every field corresponds 1:1 to an argument of the script’s main parameters. Most primitive types in Python or Typescript have a natural corresponding type in JSON so the conversion is always what you would expect. We then execute the script inside a new sandbox and then store the results in the same Postgres DB at the end of the job execution.

The HTTP payload can be sent from your own frontend or you can use our automatically generated UI. Indeed, we do a simple, yet effective analysis of the parameters of your script, and from it, generate the jsonschema corresponding to your parameters. That schema is what enables us to convert any script into a no-code like module for flows, or a standalone internal app with its auto-generated UI. In the case of Python, we also look at the imports to deduce the Pypi dependencies without you having to declare them.

For flows, we defined an open spec for building them out of those scripts we call OpenFlow: https://docs.windmill.dev/docs/openflow. It is essentially a json format for describing a sequence of steps with for loops and soon branching. The most interesting bit here is that each input of each step can define its input as a javascript expression that refers to and transforms the output of any previous step. We make it fast by leveraging native v8 integration in Rust (thanks to the deno team) for executing those expressions. This makes this apparently linear sequence a flexible DAG in which one can express complex workflows.

Then on top of that we have an UI builder for flows that hides most of the complexity to give an experience that is similar to a low-code platform where every step is treated as a blackbox. The platform itself offers all the features that you would expect: a variable and object store for storing states, plain values and credentials; a cron scheduler, tight permissioning for the sensitive credentials, groups, a webeditor with smart assistant to edit the scripts directly in the platform etc. Finally, we made a hub (https://hub.windmill.dev) to share flows and scripts with everyone. The goal is to grow over time an exhaustive library of pre-made modules and flows to tweak from so that you can focus on what is actually custom to you.

Windmill is open-source and self-hostable. You can think of it as a superset of both Pipedream and Airplane.dev. Compared to Temporal, the scripts themselves are agnostic of the flow in which they are embedded, which has the benefit of making it easier to build a hub of reusable modules. We are the only ones as far as we know to convert script parameters to UI automatically. We see ourselves as complementary to UI builder solutions like Retool or Tooljet as we do not want to focus too much on the auto-generated UI and could be used solely as the backend part of the two aforementioned tools.

We are now a team of 3 senior engineers and the product is progressing faster than ever with a public roadmap: https://github.com/orgs/windmill-labs/projects/2

We make money from commercial licenses, support and team plans on the hosted solution.

You can self-host it or try it https://app.windmill.dev, the free tier is generous (and the paid one is not enforced yet). Our landing page is: https://windmill.dev. We would appreciate your feedback and ideas and look forward to all your comments!

user:	lveillard
created:	September 22, 2021
karma:	24