Show HN: Conducto: Next-Gen CI/CD in Python, Not YAML

Show HN: Conducto: Next-Gen CI/CD in Python, Not YAML(conducto.com)

20 points by jonsolo 5 years ago | 7 comments

jonsolo 5 years ago |

Hey everyone, I'd like to announce the launch of Conducto and its CI/CD tool. As a co-founder, I originally developed the technology to do data science at my previous employer, Jump Trading (top quant trading firm like Citadel and Jane Street). As we built it into Conducto, we realized how good it was for our own CI/CD and switched course to target that first. Main distinguishing features:

* CI/CD in Python, not YAML: simple and powerful way to define CI/CD pipelines that is understood by IDEs

* Trees not DAGs: DAGs are messy. If you've ever traced a node's dependencies in a medium-sized CircleCI project (speaking from experience here) you know how frustrating it is to search through a 2000-line YAML file. Trees help your pipelines stay simple as you grow, and they're much more useful to look at.

* Free mode that doesn't suck: Run pipelines on your laptop for free, forever, for you and as many teammates as you want. When you need more scale you can run hundreds of tasks simultaneously in the Conducto cloud.

We're launching today, and I'd like to invite you all to check it out. I want feedback as much as I want users, so please look at it and let me know what you think.

dpzmick 5 years ago |

I'm another user of the predecessor to Conducto. We're using it to orchestrate very large regression test (usually ~40-50k independent processes run as part of the regression testing).

I also use conducto's predecessor to manage "data science" workflows (simlar to https://www.conducto.com/docs/advanced/data-science) on a large computer. It is miles ahead of the alternatives we've tried (and built).

The biggest win we've gotten with conducto comes from the composability. We can ship part of our application as a function which returns conducto nodes. Someone else can call this function an stick it into their own tree (code, not configuration!)

Our integration test suite builds a conducto tree ships containing test code from 4ish different libraries (developed by 2 different teams), each of which includes it's own tree-building functions. Likewise, we ship many of our applications for the large computer as functions which return conducto nodes. Users just call our functions and embed our portion of the workload into their own trees.

Give it a shot!

detaro 5 years ago |

With that tagline, an obvious comparison would be to buildbot. Any specific points in regard to that?

jonsolo 5 years ago | |

Sorry, I'm just coming up to speed on BuildBot in response to your question. There are a lot of tools out there, so apologies if I am misinformed.

Okay. From my quick reading, BuildBot is a nice way to run your tests in parallel. It also has Jenkins-like dependencies, where a successful build can trigger another build.

In Conducto, these parallel and serial steps aren't broken apart, they are combined into a single tree. The tree lets you compose these steps to do bigger and more sophisticated tasks, all within a single build. Check out https://medium.com/conducto/cant-teach-an-old-dag-new-tricks... for more details.

This makes it easy to chain together steps like "Build my artifacts", then "Test them", then "Deploy my app", where each of the steps has more detail.

Also, Conducto has a big focus on debugging that I don't see in BuildBot. In my experience, debugging new features is a major productivity bottleneck for developers, and we built Conducto to make it easy to reproduce errors and solve them quickly, even in long-running pipelines.

I'll keep reading, these are my first thoughts.

coutud 5 years ago | |

Having used conducto's predecessor extensively, I'd say the approach is more generic.

Buildbot seems very specifically geared towards a CI/CD type of workflow (like jenkins/buidbot), with the classic checkout/build/test/deploy(/notify) steps (cf https://docs.buildbot.net/current/manual/introduction.html#c...). Buildbot seems to allow you to customize some steps within the build part, that will run on the same worker.

On the other side conducto is meant to allow you to describe your workflow in python, without forcing you to follow a specific structure. It's up to you to decide how to parallelize things (I can think of build examples where conducto is used to build many objects (ParallelNode), then link). You just define the successsion of serial/parallel tasks that are needed in your automation, rather than trying to make it fit in whatever the makers thought was a proper way to build/test/deploy.

SEJeff 5 years ago |

“Configuration as code, not YAML” - this is a hilarious and relatable tagline.

I wonder how this compares to some existing CI/CD tools like Jenkins, circle ci, GitHub actions, etc.

jonsolo 5 years ago | |

Conducto vs CircleCI: https://www.conducto.com/docs/comparisons/conducto-vs-circle...

GitHub Actions is like a better version of CircleCI. They really cloned a lot of it and improved it. Similar config YAML, similar UI. The points in the CircleCI article largely hold, except for the free mode: GHA does have local runners that are very similar to Conducto's local agent.

Jenkins comment coming shortly.