Cortex – An ML model deployment platform that runs in your AWS account

Cortex – An ML model deployment platform that runs in your AWS account(github.com)

65 points by ospillinger 6 years ago | 24 comments

sandGorgon 6 years ago |

You should be using Terraform as your configuration management language. Please do not invent your own. Pretty much your yaml is a pseudo-configuration management language.

That will become one of the biggest blockers because of the amount of automation that already exists.

By leveraging Terraform, you will also have the added benefit of getting all the other pieces of AWS/GCP/azure components for free - and is rock stable and production tested.

ospillinger 6 years ago | |

Thanks for the feedback! We aren't trying to invent another infrastructure provisioning language, and I agree that Terraform would be the right choice if that was the case. Our YAML is more similar to the configuration of deployment tools like Netlify or CircleCI. We use CloudFormation and Kubernetes under the hood but our goal is to provide a much higher abstraction for data scientists / ML engineers.

sandGorgon 6 years ago | | |

Not entirely. The abstractions are different between infrastructure deployment.... versus configuration yml of circleci.

The declaration of deployment state is a very BIG and hard problem that has had millions of collective man hours spent over decades. I urge you not to think of it as a simple configuration.

In fact it is so hard that AWS has to build a new language on top of typescript ..versus cloudformation templates that it already had.

https://docs.aws.amazon.com/cdk/latest/guide/home.html

What you are building makes sense - I would drop cloudformation and surface Terraform right till the top.

So the way to use your tool is to install and use a new Terraform "provider".

fake-name 6 years ago | |

You should be using an existing scripting language as your configuration language.

Seriously, Every single fucking stupid infrastructure-deployment-tool/"platform" whatever has it's own, dumb in-house language that winds up basically re-implementing the programming language the tool is written in badly.

  - Puppet: Has a stupid ad-hoc config language.   
  - Terraform: Has a stupid ad-hoc config language.   
  - SaltStack: Has a stupid ad-hoc config language.   
  - Ansible: Has a stupid ad-hoc config language.

If you're even considering implementing a tool like this, use a goddamn existing language for your configuration files.

You don't need to use the entire language, but at least use the language's lexer/parser (cf. json/javascript). That way, all existing tooling for the language will work for the config files (ask me about how saltstack happily breaks their API because you're not "supposed" to use it, despite the fact that they have public docs for it). Additionally, people won't need to figure out all the stupid corner cases in your terrible piecemeal configuration language.

Additionally, by making your configuration language an actual language, you also simplify a lot of the system design, because the configuration can act directly against your API. This means using your tool from other tools becomes much more straightforward, because the only interface you actually need is the API.

sandGorgon 6 years ago | | |

You are right - Terraform does have the ecosystem, but the new kid on the block is Pulumi.

Pulumi = Terraform in Typescript. That's good as well - but i was not sure if the OP is familiar with Pulumi

icedchai 6 years ago | | |

Yes!! Glad someone said this. Most of these languages, Terraform "HCL" specifically, are awful for anyone familiar with a real language.

mlevental 6 years ago | | |

https://github.com/cloudtools/troposphere

personjerry 6 years ago |

Man I can't way to pay for a ML dev service that handles the infrastructure dev service that I pay for to host my ML dev services.

tixocloud 6 years ago | |

We're actually building infrastructure to serve ML as a Saas and would love to get your input.

metabeard 6 years ago | | |

whoosh ;)

scribu 6 years ago |

A comparison with AWS's own offering (SageMaker Model Serving) would be helpful.

ospillinger 6 years ago | |

We solve a similar problem to SageMaker but we are focused on developer experience and flexibility.

- Deployments are defined with declarative configuration and no custom Docker images are required (although you can use your own if you want)

- You have full access to the instances, autoscaling groups, security groups, etc

- Less tied to AWS (GCP support is in the works)

- We are working on higher level features like prediction monitoring, alerting, and model retraining

- It's open source and free vs SageMaker's ~40% markup

jmcminis 6 years ago |

Compare and contrast with Seldon on Kubeflow?

ospillinger 6 years ago | |

My understanding is that Seldon and Kubeflow are more geared towards infrastructure engineers. Our goal is to hide the infrastructure tooling so that Kuberentes, Docker, or AWS expertise isn’t required. Cortex installs with one command, models are deployed with minimal declarative configuration, autoscaling works by default, and you don’t need to build Docker images / manage a registry.

jmcminis 6 years ago | | |

Thanks!

I bet you could get Cortex running on Kubeflow pretty easily since it's all K8s anyway.

hamolton 6 years ago |

Has anyone used this? It looks useful!

- Puppet: Has a stupid ad-hoc config language. - Terraform: Has a stupid ad-hoc config language. - SaltStack: Has a stupid ad-hoc config language. - Ansible: Has a stupid ad-hoc config language.