How to rewrite it in Rust(adventures.michaelfbryan.com) |
How to rewrite it in Rust(adventures.michaelfbryan.com) |
By rewriting small components and integrating them into the existing project using Javas native interface, our small team of 5 developers were able to pull off this massive rewrite in just under a year. The resulting code base is rearchitected in a few very important ways, but mostly follows the same structure.
And because we kept and evolved our old Scala based test suite, we have a very high confidence in the rewrite.
When Async/.await finally landed, we could switch over very quickly, and it has been a joy to focus on benchmarks and performance over the last month. Spoiler: Rust is faster than Scala :-D
I ask because Scala already has a good type system and the JVM typically has good performance nowadays, particularly with something like GraalVM, so I am actually really curious to why you felt a Rust rewrite was a good idea.
* you can keep memory use quite a bit lower * you can still sometimes get large constant factors of performance improvements over the JVM in some kinds of problem domains. If this means you can run on 1 server instead of say, 5, you have a much simpler infrastructure. * startup time, especially if you are doing 'serverless' or similar * tail latency - even a good GC language will have occasional long pauses. * data race protection at compile time * easier deployment - no need to install a jvm and keep it updated
Huge shoutout to everyone working on Rust Async, especially the Berlin crew, who has been very helpful.
I guess the plan what to make native is pretty important. Maybe you want to disclose more details about that?
And the old Scala codebase here: https://github.com/prisma/prisma/
The old codebase has parts in Rust, so counting the lines is not that straightforward.
You don't actually have to rewrite everything on day one. You can stop writing new C code right now, and then gradually replace old code only when you need to fix it or refactor it.
As we see below, you may still need to write some code to convert from C types to something that is more ergonomic to use in Rust. But the marshaling ABI-wise is minimal.
> Turns out the original tinyvm will crash for some reason when you have multiple layers of includes
It's actually crashing because there are no lines of code in the file, so certain data structures never get initialized.
Exactly, as much as I love writing Rust code, there is nothing that is more frustrating that maintaining bindings from C to Rust. Then you'd have to create another idiomatic binding on top of it. Sure bindgen is great, but Zig's cImport and Swift's ClangImporter take this further.
They both use clang modules to tackle the first part. A autogenerated idiomatic solution would be great for Rust, but sadly it doesn't exist.
Yikes - port the entire build system to cargo before you write a line of Rust. Now draw the reset of the owl!
Surely's there's an incremental path for the build as well? Perhaps if you're using CMake?
Moving C build to build.rs is not necessary. It's usually done only because people used to Cargo don't like bringing CMake along. If you were to publish this as a Rust crate, it'd be slightly easier for downstream Rust users to have one less external tool to install.
The big difference is however that JS and TS can live side by side on a file by file basis out of the box with the Typescript compiler, which makes it super easy to convert. You don't have this luxury with C and Rust of of the box, but serious kudos to the author for finding a way to do something very similar.
When converting C to Rust you usually have to do things on a module by module or compile artifact by compile artifact basis, which makes it much more challenging. You can however employ some sort of strangler pattern: https://docs.microsoft.com/en-us/azure/architecture/patterns...
OP is essentially about proving the opposite. It does take a bit of setup to get there, but you can ultimately translate C to Rust on a function-by-function basis, and Rustify interfaces, data structures, etc. only gradually after nothing on the C side is relying on the older defs.
C++ would be more of a challenge - you need to forgo quite a few C++-exclusive features to end up with interfaces that Rust can work with. That's where an "artifact by artifact" approach might work better. Other languages would be roughly similar, with their heavyweight C FFI's.
Annoyingly, at least with the typical msbuild pattern, you have to be using the same language at the project level, but you can have as many projects as you want per solution (it's weird). So it's not as seamless as the JS/TS system, but overall it's not too bad, since you still can mix and match somewhat.
Luckily, many C++ projects do this already so they can be called from C.
Regardless, major kudos for your rewrite!
It wasn't just me; it was everyone on my team, and probably everyone in the company; I'm pretty glad they did that though; F# ain't perfect, but I like it a lot better than C#.
On the long pauses, I've built simulation server systems for multi-user in .Net and the stop the world GC for several seconds now and then was very painful practically speaking.
We distribute the binary with our NodeJS package and hundreds of megabytes of binary size will not work that well for our users.
Thanks for the insight!
What we did in the end is we rewrote parts of the system with Rust, plugged that to the JVM package and we had all our tests ready to use. First the database connectors and at the same time the other part of the team was writing the graphql parsing in Rust. We could all utilize our Scala integration tests which was crucial for our success.
And btw. we still have our tests in JVM, although the rest of the stack is Rust now.