The goal of Carton is to let you use a single interface to run any machine learning model from any programming language.

It’s currently difficult to integrate models that use different technologies (e.g. TensorRT, Ludwig, TorchScript, JAX, GGML, etc) into your application, especially if you’re not using Python. Even if you learn the details of integrating each of these frameworks, running multiple frameworks in one process can cause hard-to-debug crashes.

Ideally, the ML framework a model was developed in should just be an implementation detail. Carton lets you decouple your application from specific ML frameworks so you can focus on the problem you actually want to solve.

At a high level, the way Carton works is by running models in their own processes and using an IPC system to communicate back and forth with low overhead. Carton is primarily implemented in Rust, with bindings to other languages. There are lots more details linked in the architecture doc below.

Importantly, Carton uses your model’s original underlying framework (e.g. PyTorch) under the hood to actually execute the model. This is meaningful because it makes Carton composable with other technologies. For example, it’s easy to use custom ops, TensorRT, etc without changes. This lets you keep up with cutting-edge advances, but decouples them from your application.

I’ve been working on Carton for almost a year now and I’m excited to open source it today!

Some useful links:

* Website, docs, quickstart - https://carton.run

* Explore existing models - https://carton.pub

* Repo - https://github.com/VivekPanyam/carton

* Architecture - https://github.com/VivekPanyam/carton/blob/main/ARCHITECTURE...

Please let me know what you think!

user:	gemaif1li
created:	September 28, 2023
karma:	1