Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects? We are working on an ML project (Computer Vision specifically) and we are in the process of productionizing our models. What kind of tools do you use and what's your workflow for integrating, testing and deploying your models? Do you have any suggestions or tips about what to do and what to avoid? |
First, I would highly recommend wrapping your ML models in some kind of microservice. Depending on your production requirements and if the ML is in Python a fairly simple Flask/Sanic web server should be sufficient. This is great because you can leave all your feature transformation code as is in Python.
If your production environment has very low latency requirements you are going to have some work cut out for you. You'll most likely have to rewrite all your transformation code in a faster language like Go or Java. You might also need to implement the inference code as well to get the speed you need. This adds considerable time and adds a ton of surface error for potential insidious bugs. The ML will still make predictions, but they will be wrong or very slightly wrong.
Because I'm working with larges of amounts of data and my source of truth is Parquet logs in S3, the pipelines start with Spark. We do as much data wrangling as possible in Spark to get things into a manageable size to create our train/dev/test sets. This data gets uploaded to S3.
The datasets are then trained on EC2 instances using Pandas & sklearn. When everything is fully automated the Spark job will push a message onto an SQS queue with the S3 path of the fresh dataset. An EC2 instance will be polling that queue and pull down the data and train a new model.
The final result of training my case is a text or binary model file that goes back up to S3. Our prediction microservice polls an S3 bucket and pulls down any updated model files and swaps out the running models.
Tips:
1. Instrument everything! Hopefully you have something like graphite/datadog/prometheus in place already, but you'll want metrics on your predictions.
2. Exception tracking on everything especially anything in your model creation pipeline. Sentry or something like that.
3. Try and keep everything as simple as possible.One piece did give me a bit of surprise:
You might also need to implement the inference code as well to get the speed you need
Also like GP, I used to work in advertising. The low latency bits are because RTB servers want you to respond within a certain amount of time. When I was in advertising, our solution was to precalculate a whole bunch of things, and throw them into redis. It was then a simple lookup of the hash of a vector (for bidding related stuff)
The constraints force us to use simple models like linear regression and logistic regression some of the time or at least as a version 1. The inference here is straightforward, multiply and add then take the sigmoid if doing logistic regression.
What we tried to do initially was integrate with C/C++ APIs where possible. We ran into some issues with speed and bugs doing this though, which is why we wrote the inference ourselves. Where we had issues was calling the XGBoost C API from Go. It was extra overhead and too slow. In our benchmarks our implementation in pure Go was many times faster than calling the C API. We also found the multithreaded version to be slower than the single threaded. We found this to be true when calling XGBoost from Java and from Go. We also found this to be true in our own inference implementation it was always faster to walk the trees in a single go routine rather than create some number of worker go routines to walk the trees in parallel.
We were very careful implementing the inference ourselves to make sure the predictions matched. What we did to verify this was create a few toy datasets of about 100 rows with sklearn's make_classification function. We then trained a model using the reference implementation, saved the predictions and the model. We then loaded this model into our implementation and made predictions on the same dataset. We wrote unit tests to compare the predictions and make sure they are the same within some delta. We were able to get our implementation to be within 1e-7 of the reference implementation, in this specific case XGBoost. It was actually more time consuming to deal with parsing the inconsistent JSON model output of XGBoost than it was to implement the GBDT inference algorithm. We also had to make a slight change to the XGBoost code to write out floats to 18 decimal places when writing out the JSON model in order to get the two implementations to match.
Definitely lots more options than I even had space to cover in a single book, though. This is an area where the tooling continues to rapidly improve.
[1] https://engineering.semantics3.com/moving-machine-learning-f...
1. Keras/Tensorflow based algorithm(Applicable for any compute intensive or GPU-capable algorithm): Deployed the method (as a flask service) inside a Docker container along with a queueing system(for reliability w/ redis). We can now decide the server type and the kind of orchestration tool we can use to manage these containers. Following are some options for it,
a. ECS on AWS
b. Kubernetes
c. Docker Swarm
d. custom orchestration tool
Tips:
i. Have the memory flush in the code after the service is used so that there is no memory leak.
ii. You can use tools such as htop to understand the memory usage.
iii. Regarding system performance, you can use prometheus to gather stats along with grafana dashboard to view them.
1) training code is written with normal packages (we tend to prefer keras with a tensorflow backend), these are trained on our own GPUs since this is often the cheapest and are done in bulk --> side note is that TPUs/GPUs may one day be better but certainly too expensive currently.
2) prediction / inference code will also be written in the similar packages but will be tied with the final weights files that we get from the model
3) deployment code -- in order to enable a reliable system we have used celery for distributed queueing and converting our prediction functions into celery tasks which can then me passed to workers that can process and return the result to an API endpoint. This allows us to scale our workloads as needed with throughput (depending on request requirements)
This architecture allows us to test during training time using our training code and validation sets, while also enabling testing of different models versions through our prediction APIs. We often would just write scripts for testing that we then run via our CI/CD workflow.
Tip: keep your code as simple as possible and don't rewrite your code unless your throughput requirements mandate it. good error handling will go a long ways here and likely will be easier if written in a language you're most familiar with.
1. all lowish-level production code (importing, transforming, modeling) is written in packages, which are thoroughly unit-tested with a CI system
2. that code is wrapped up into docker container(s) in separate repositories, which are built and integration-tested with CI. In addition to the Dockerfile and any testing scripts, there's usually a single code file here which handles I/O specifics, API endpoints, and primarily calls code from the package
This works well with R or Python and should work with others; we use Gitlab for the free private repos and awesome built-in CI.
This doesn't cover where the data or models are stored, but that varies more per-project for us. Lately we've been using Pachyderm and loving it, but you can get pretty far with a postgres instance for data and storing trained model objects in S3/GCS.
Our system streams in market data from exchanges, creates forecasts with python/sk-learn and displays the data, we also have background processes that updates our accuracy over time once real data is available.
We test our code with the normal python unit/integration/end to end testing methods locally with local copies of all of our components (except firebase for live UI updates, we don't have a dev version of that yet just use live forecast data when testing the UI charts/display), would probably get expensive/cumbersome to setup dev environments with local firebases in them.
with deployment we simply ssh into machines, git pull latest code and supervisorctl restart so its fairly low tech, the forecaster has a roughly minute outage when we deploy new models because there is a decent process that computes and caches a data structure of historical features for use in the forecaster.
In terms of maintaining a reliable online stream of data input, feature computation/prediction pipeline we run the code under supervisor aswell as running a manager process under supervisor, that manager process checks if conditions are turning bad (OOM/no progress updates by the forecaster) and restarts things if anything goes wrong.
For testing we also use the standard training/test data split when running backtesting/machine learning optimisation algorithms to train parameters of the algorithm. If things perform better on training and test data over a long enough time period to build confidence then we will deploy a new model.
Using graphite/graphana to monitor prediction accuracy over time is a good idea as mentioned already :) and some kind of alerting/monitoring if things go down.
As far as putting into production I'm not as familiar. Yet I hear that a good Python workflow would probably work best.
The killer features for me are nested subgroups (which bitbucket may have, but github does not) and a really awesome CI system with a generous free tier (2000 minutes/month). For R packages, we have it setup very similar to github + travis (devtools::check() every push), and for deployable bits we have it build containers and run integration tests on them. Super impressed with all we get for free there.
Also per the production environment. The key is having the same machine learning libraries available in both the development and production so you can plug the model in with little problems with dependencies. For this reason most folks that will go into production, particularly web applications, will tend to both develop and implement in Python, Java, etc.
FWIW: Production is an overloaded term. CI may not even be applicable here. Say you're doing batch inference where you need to run jobs every 24 hours on a large amount of data: That might be tied to some cron job.
That being said you could use a CI system for that in theory.
There are also other factors here: What other kind of things do you want to track? Experiments results? Wrong results by your machine learning algorithm?
Concisely: What kind of deployment requirements do you have and what are your goals?
If you are edoing real time, what does "deployment" even mean? Are you serving in real time via a rest api? Are you doing streaming? What are your throughput requirements? What about latency? Is that even hooked up to a CI system?
Something that is vaguely related: How do you test the accuracy of different models across your cluster? Say you want to do a self deployment, what if you want to tie that to say: a workspace where you produced the results?
Is that hooked up to a CI system? If so, what's your use case?
Then there's that common hand off from data scientist to production, what does that look like? A sibling thread mentioned some of these things.
If anyone else is curious about this stuff, we deploy deep learning models in locked down environments both on kubernetes as well as touching hadoop clusters. Happy to answer questions.
[1]: https://algorithmia.com/tags/computer-vision [2]: https://algorithmia.com
It does look like a cool service, though!
Part of me is thankful to not have these problems, while another part thinks it'd be a lot of fun to do this kind of last-mile engineering.