An update on Dart macros and data serialization(medium.com) |
An update on Dart macros and data serialization(medium.com) |
I'm bummed that it's canceled because of the lost time, but also relieved that we decided to cancel it. I feel it was the right decision.
We knew the macros feature was a big risky gamble when we took a shot at it. But looking at other languages, I saw that most started out with some simple metaprogramming feature (preprocessor macros in C/C++, declarative macros in Rust, etc.) and then later outgrew them and added more complex features (C++ template metaprogramming, procedural macros in Rust). I was hoping we could leapfrog that whole process and get to One Metaprogramming Feature to Rule Them All.
Alas, it is really hard to be able to introspect on the semantics of a program while it is still being modified in a coherent way without also seriously regressing compiler performance. It's probably not impossible, but it increasingly felt like the amount of work to get there was unbounded.
I'm sad we weren't able to pull it off but I'm glad that we gave it a shot. We learned a lot about the problem space and some of the hidden sharp edges.
I'm looking forward to working on a few smaller more targeted features to deal with the pain points we hoped to address with macros (data classes, serialization, stateful widget class verbosity, code generation UX, etc.).
Due to AOT compilation, some form of (pre)compile time code generation is needed, but it doesn't need to be macros. It doesn't need to be instantaneous, but it also shouldn't take minutes.
Adding features directly into the language removes the need for some code generation.
Augmentations will already make code generation much nicer to use.
build_runner needs to become more integrated so that IDEs would read build_runner's config and run it automatically.
Though it would not surprise me at all if it was considered too slow to adopt. I don't remember compile-time performance issues with it when I did Java work, but I was definitely not paying close attention, and the ecosystem strongly prefers runtime shenanigans in general.
In practice in a Dart app you usually use freezed or something similar: https://pub.dev/packages/freezed
When the language implementors start making larger programs, it will soon become apparent how the program organization is hampered without named, defined data structures.
I didn't add structs to TXR Lisp until August 2015, a full six years from the start of the project. I don't remember it being all that much fun, except when I changed my mind about single inheritance and went multiple. The first commit for that was in December 2019.
Another fun thing was inventing a macro system for defstruct, allowing new kinds of clauses to be written that can be used inside defstruct. Then using them to write a delegation mechanism in the form of :delegate and :mass-delegate clauses, whereby you can declare individual methods, or a swath of them, to delegate through another object.
i.e. I annotate my models with @freezed, which also affords you config of the N different things people differ on with json (are fields snake case? camel case? pascal case?) and if a new N+1 became critical, I could hack it in myself in 2-4 hours.
I'm interested to see how this'd integrate with the language while affording the same functionality. Or maybe that's the point: it won't, but you can always continue using the 3rd party libraries. But now it's built into the language, so it is easier to get from 0 to 1.
There are a bunch of other interesting approaches here they can look at. Improving the code generation story more generally, shopping the augmentations feature (basically C#’s partial classes) and getting more serious about serialization all feel like sensible directions from here.
There is a really interesting community proposal at the moment on the serialization front that I think would solve a lot of the issues that got people so excited about macros in the first place here: https://github.com/schultek/codable/blob/main/docs/rfc.md
It's a big bite to chew, but I think Roslyn has paid big dividends.
Does anyone have any more information on How Dart actually does Tree Shaking? And what is "Tree Shakeable"? This issue is still open on Github https://github.com/Dart-lang/sdk/issues/33920.
I think this quote accurately sums things up
> In fact the only references I can find anywhere to this feature is on the Dart2JS page:
> Don’t worry about the size of your app’s included libraries. The dart2js tool performs tree shaking to omit unused classes, functions, methods, and so on. Just import the libraries you need, and let dart2js get rid of what you don’t need.
> This has led customers to wild assumptions around what is and what is not tree-shakeable, and without any clear guidance to patterns that allow or disallow tree-shaking. For example internally, many large applications chose to store configurable metadata in a hash-map:
However, this is a topic of active work for the Dart team: https://github.com/dart-lang/build/issues/3800. I'm sure they would welcome your feedback, particularly if you have examples you can share.
You're also always welcome to reach out to me if you have Flutter/Dart concerns. I founded the Flutter project (and briefly led the Dart team) and care a great deal about customer success with both. eric@shorebird.dev reaches me.
It may be worth mentioning that build_runner's graph contains every single asset that might be generated. So when selecting what's included and excluded you can reduce the graph size dramatically.
I'd like to believe this is a good thing for the Dart project, but only time will tell. My hot take here: https://shorebird.dev/blog/dart-macros/
In my humble opinion, you can handle many cases like serialization better with 'compileTime' or comptime features though I'm partial to macros. Especially with core compile time constructs like 'fields' [1, 2]. Though those require some abilities dart's compiler may not have or be able to do efficiently. That'd be a bummer, as even C++ is finally getting compile time reflection.
1: https://nim-lang.org/docs/iterators.html#fieldPairs.i%2CT 2: https://www.openmymind.net/Basic-MetaProgramming-in-Zig/
The big metaprogramming feature traditionally implemented in macros, type generation, is already provided in some form by all major languages already.
And an awful lot (and I mean an awful lot) of good work can be done at the string replacement level with cpp. And generating code upstream of the compiler entirely via e.g. python scripts or templating engines is a very reasonable alternative too. And at lower levels generating code programmatically via LLVM and GPU shaders is well-trodden and mature.
Basically, do "macros" really have a home as a first class language feature anymore?
Yes macros can be a pain and should be limited, but in my experience, a couple hundred lines of macros replaces many thousands of lines code generators with complicated baroque build system integrations (ahem ROS2). The tradeoff is even worse when the language supports templates and compile time operations which can usually replace macros with even less code and are easier to understand. Though at least Go supports codegen properly with support in its official tooling.
1: https://github.com/google/flatbuffers/blob/master/src/idl_ge... 2: https://github.com/python/cpython/issues/94675
Language execution speed isn't the fundamental blocker for code generation in Dart. Dart isn't quite as fast as C++ or Rust, but it's in roughly the same ballpark as other statically typed GC languages like C#, Java, and Go.
The performance challenges in code generation are more architectural and are around cache invalidation, modularity, and other tricky stuff like that.
Without investing significant time, like they did with Dart, they would have a language with a much bigger ecosystem that is faster, already has compile time code generation and better support for data than Dart. It supports ahead-of-time compilation and hot-reload. The only feature missing in C# is compilation to JS, but with WASM is that really needed? Biggest downside of C# is probably that it's not invented at Google.
They must have done something wrong. Macros are expanded when you ahead-of-time compile your code, which doesn't take place in the run-time environment where you hot load, but in the build environment. It doesn't matter whether the macro are simple, or whether they can inspect lexical environments and look up type info and whatnot.
Compile-time costs should never factor into hot reload, because the stuff being loaded should already be compiled.
Maybe they aren't explaining it; there could be certain semantic problems preventing existing state from being re-used on what should be a hot reload.
Macros create certain issues in reloading. If you change a macro such that the expansion requires different run-time support which is incompatible with existing expansions, you have problems. One option may be to reload all the code which depends on those macros, so that everything cuts over to the new run-time support. If you need to support a mixture: hot-reloaded modules using the new versions of the macros, side by side with code made using the old versions, then the old version of the run-time support has to coexist with the old.
If the run-time support for the macros is something which manages state that needs to be preserved on reloads, then that can cause difficulties. The old and new macro expansions want to appear to be sharing the same state, not different silos.
Dart has multiple layers where it does tree shaking.
The first one is when building the "dill" (dart intermediate language) file, which is essentially the "front-end" processing step of the compiler which takes .dart files and does amount of processing. At that step things like entire unused libraries and classes are removed I believe.
When compiling to an ahead of time compiled binary (e.g. for releasing to iOS or Android) Dart does additional steps where it collects a set of roots and walks from those roots to related objects in the graph and discards all the rest. Not unlike a garbage collection. There are several passes of this for different parts of the compile, including as Dart is even writing the binary it will drop things like class names for unused classes (but keep their id in the snapshot so as not to re-number all the other classes).
I have no experience with tree shaking in the dart2js compiler, but there are experts on Discord who might be able to answer: https://github.com/flutter/flutter/blob/master/docs/contribu...
What exactly all this means as a dev using Dart, I don't know. In general I just assume the tree shaking works and ignore it. :)
The Dart tech lead has done some writings, but none seem to cover the exact details of treeshaking: https://mrale.ph/dartvm/ https://github.com/dart-lang/sdk/blob/main/runtime/docs/READ...
The point is really that "macros" is a weird sandwich between "complicated metaprogramming you need to do from first principles" and "you really didn't need metaprogramming, did you?". And that over the years that sandwich has been getting thinner.
Lisp macros in the 60's were a revelation. They don't really have a home anymore.
help
meI'm pretty sure we did look at C# (and certainly a whole bunch of other languages). I don't actually recall why we didn't use C# at the time. I remember Go binaries were waaay to big, JS (what we originally wrote Flutter in) startup time was way too slow on iOS, Swift was too deeply tied to Apple (the standard library was closed source at that time), etc. It's possible that C# was too verbose or didn't have a path to hot reload? But that's just a guess. I'm not a C# expert, and Adam Barth drove most of the language evals at the time.
That said, I'm also not sure Miguel (creator of Xamarin) would agree. He's a Flutter fan now (and backer of Shorebird, my company).
Past discussions: https://x.com/migueldeicaza/status/1778759403451081159 https://x.com/migueldeicaza/status/1559898665350832128
Anyone who worked with the mobile .NET and Flutter would see Dart/Flutter DX as something unreachable for .NET, it's a terrible experience like any other .NET cross compilation I've tried (Blazor, Silverlight).
I'm not a big fan of Dart as a language but it really was a great choice that allowed amazing DX, Flutter hot reload feelt better than JS/HTML.
That was the same year that MS first released C# core. For cross platform support mono was really the only way to go and it was second class.
MS was just starting to get out of the mindset of putting the universe into the .Net framework and instead offering first class support for a broader 3rd party ecosystem.
How Microsoft operates today with open source software really started roughly around 2016. I could see why you'd be hesitant to trust them then.
I think folks forget how long ago we made Flutter.
Despite the "not invented at Google" swipe
* TypeScript wasn't invented at Google either, but adopted heavily.
* Angular was the first major project ever to use TS.
* And interestingly enough, Anders Hejlsberg contributed heavily to both.
He's very complimentary of the goals and governance of Flutter, which is certainly more important than a language choice between two respectable languages.
I do think C# is by far the best mainstream language, but good IDE support and library ecosystem are the dealmakers/breakers for me when choosing a stack for a project.
Dear lord no. We don't need more C# in the world.
>It supports ahead-of-time compilation and hot-reload.
In name only. Doesn't really well in practice. Go and just look for "C# hot reload not working" in any search engine and look at the variety of contexts it just simply does not work with no resolution.
It shares a lot of language constructs with TypeScript (and by extension, JS) and has been converging with each release so I'm often surprised that people hate on it or that more startups don't reach for it if they are on Node with TS.
Same syntax for key language constructs like async-await, try-catch-finally, generics, etc.
Hot reload works pretty well (at least in the contexts that I use .NET (backend APIs)); a lot of the issues were from the early days. `dotnet watch` has been very much usable for the last few years.
I'm sure C# is too.
I've been unlucky enough to have many years in on both iOS and Android, and Dart is a fantastic language, far better than both incumbents.
I worry about judging it as a whole, based solely on their ability to launch pre-compile time code generation that is faster than their current approach.
Macros seemed really cool + really difficult to improve past the current codegen.
I have a 35K LOC "main" code base that generates 670K lines of code under the current approach. It takes 52 seconds for a cold generation of all 670K. Seconds for warm. shrugs (sounds great to me)
I know that the build_runner authors are looking into perf as we speak, and I'd be happy to put you in touch with them if you'd like to speak with them about debugging your case: https://github.com/dart-lang/build/issues/3800
eric@shorebird.dev reaches me (for this or any other Flutter/Shorebird issue).
Google by contrast isn't nearly as invested in Dart as Microsoft was (and still is) in C#/.NET. Perhaps a better objection is that they should have just used Go — or a Go-binary-compatible language built on some of the same toolchain. (See also: Vala and Guile still don't play nice together as well as they should for two languages from the same project.)
The user experience with hot reload is:
1. They hit "run".
2. The compiler compiles the app.
3. The app starts running on their device.
4. They change some code in their IDE.
5. They click "hot reload".
6. The compiler compiles the changed code.
7. The IDE sends the updated code to the running app.
8. The runtime loads the changed code.
9. They see the changed behavior in their running app.
Steps 6-8 determine the total time between "user requests a hot reload" and "user sees their updated app". Compilation doesn't happen on the device, but it still takes time and is in the critical path for that experience.
Making the compiler slower makes hot reload slower. We measure hot reload time in milliseconds, so it doesn't take much for us to consider it an unacceptable performance regression.
Secondly, any compilation delay they experience affects all their iterative development scenarios, including a complete application restart for each run.
If they wrote the macros themselves that are slowing down compilation that much, it is their self-inflicted problem.
Even if macros are slowing down compilation noticeably, unless you change the macros such that everything that uses them has to be recompiled, you still have the benefit of incremental compilation and hot reloading. E.g. recompiling one just one file-with-macros out of hundreds that don't get recompiled.
> We measure hot reload time in milliseconds
It takes seconds to minutes to make the code change, but when you hit the hot-key to deploy it to the target, it's gotta compile and upload in milliseconds?
That's just a silly requirement that will leave your compiler development hamstrung.
I can't even type this comment without at times experiencing character delays that are certainly more than single digit milliseconds. :)
A conclusion like "our users require hot reloads to be milliseconds, end-to-end including compilation" deserves to be researched among the user base, because I don't suspect most devs need the times to be quite that low. They are building a program, not trying to avoid getting fragged in a multi-player shooter!
Dart is a statically typed language and we wanted macros to be able to introspect over the semantics of the code the macro is applied to, not just the semantics. For example, if a macro is generating code for serialization, we wanted the macro to be able to ask "Does the type of this field implement JsonSerializable?". Answering that means being able to look up the type of the field, possibly walk its inheritance hierarchy, etc.
It's a very different problem from just "give me a way to add pretty loop syntax".
This holds for "old" lisps. There are other options. Racket and Scheme uses "syntax objects". Syntax objects contain besides the old syntax tree also source location information and lexical information.
See for example the last part of:
https://parentheticallyspeaking.org/articles/bicameral-not-h...
[0]: imagine your colleague wrote a macro that redefines for loops because at the time, it made life easier for him.
This is like asking "what if your coworker named all errs as `ok`" so everything was `if ok { return errors.New("Not ok!!"); }`. It's possible but no one does it.
This is why `defmacro` and `gensym` in common lisp are awesome, and similarly why Go's warts don't matter. Much of programming language ugliness is an "impact x frequency" calculation, rather than one or the other.
It's also why javascript is so terrible, you run into it's warts constantly all day long.
But I don't recall seeing someone re-define `define` in real life.
Nor do I recall seeing any problematic redefinitions in Scheme in real life.
That said, if you wanted to make a language variant in which `define` did something different than normal (say, for instrumentation, or for different semantics for variable syntax), then you'd probably use Racket, and you'd probably define your own `#lang`, so that the first line of files using your language would be something like `#lang myfunkylang`.
You can randomly sample code in <https://pkgs.racket-lang.org/>, and look for people doing anything other than `#lang racket/base` or `#lang typed/racket/base`.
I'd definitely agree with you about Zephyr's "I really want C macros to be real macros" system for it's device tree system. It's a huge pain to debug because you don't find issues until the linking step. However I'd counter with the fact that C macros _aren't_ real macros. Hence the pain.
Actually C's "macros" are literally just a separate codegen tool (m4 processor) just like what you're suggesting. It's a large part of what makes C code so hard to programmatically parse or automate.
Essentially Zephyr is doing meta-programming in an external codegen tool. Totally agree that a separate codegen tool specifically for device trees or whatnot could be better than that. However if C had a proper macro system (and compile time types), you could readily express the device tree system in the language at compile time and produce helpful errors.
> The point is really that "macros" is a weird sandwich between "complicated metaprogramming you need to do from first principles" and "you really didn't need metaprogramming, did you?". And that over the years that sandwich has been getting thinner.
They aren't really though. Homebrew codegen systems are much more of the weird sandwich between "you needed metaprogramming" and "you didn't have metaprogramming". Instead you build a system which is even more complicated in total, e.g. source and generated code gets out of sync, you have name clashes, the core API logic is spread out over different languages and code bases, etc.
Though I do agree the use cases for macros are shrinking, but more due to meta-programming via templates and compile time expressions becoming more powerful and usually preferable method to doing meta-programming.
(is this the root NIH syndrome?! I'm guessing no, I'm only 36. maybe LISP enters the picture here?)
Yes, hot reload is a developer feature in Dart, not an end user feature.
> If they wrote the macros themselves that are slowing down compilation that much, it is their self-inflicted problem.
The compile time impact we saw, unfortunately, wasn't entirely linear in the amount of macro applications that a user had. If macro application time was entirely pay as you go, then, yes, it would be feasible. But it impacted compiler performance worse than that.
> It takes seconds to minutes to make the code change, but when you hit the hot-key to deploy it to the target, it's gotta compile and upload in milliseconds?
Yup! Those seconds to minutes are meaningful time well spent by the user thinking about their program and the problem. Those milliseconds are just them sitting on their thumb getting mad at the machine.
> deserves to be researched among the user base, because I don't suspect most devs need the times to be quite that low.
I would suggest to you that after working on Flutter for nearly a decade, conducting user surveys every single quarter, gathering metrics from our tool usage (opt in) and lots of other UX research, that we do have a pretty good idea of what our user base wants in regards to performance. :)
Did you just ask Flutter users? We've been using Dart for our company's stack for years, no Flutter. It seems you just took the Flutter dev experience into account. This makes this decision all the more disappointing to us.
You can't change your focus from the window where you are editing the code to the window where you are interacting with the app in milliseconds. Maybe triple digit milliseconds at best, not double, let alone single. Well, double may be within reach, if it's hot-keyed.
https://www.youtube.com/watch?v=xqGAC5QCYuQ is a talk where we discuss what led to modern Flutter (including 3 attempts in JS).
Angular was originally in Dart is my understanding, but eventually forked into two projects. Angular Dart (which is really only still used internally at Google, mostly for Google AdWords which makes all the money) and Angular JS which is what has seen so much popularity more generally.
https://github.com/facebook/hermes
https://x.com/tmikov/status/1869945330638442651
Btw that Angular history is backwards. The first versions of angular were written in JS back in ~2009. jQuery was the most popular way to build web apps, and Angular provided a (very fancy) declarative data binding framework over it. Everything ran in your browser by inspecting DOM attributes.
Later in ~2014 they built a new framework on similar principles in TypeScript, but with an AOT compiler and called it "Angular 2". Then they retconed the original framework "AngularJS" and made the 2+ framework be just "Angular". In that era some Ads folks forked the Angular 2 framework and rewrote it in Dart and the two frameworks evolved separately since.
So there's really 3 separate "angular" frameworks...
I meant to point out that you can't just assume a priori it was NIH syndrome, as Google's heavy adoption of TS is a counterexample.
Ironically, Typescript is not the best one. I can go into great detail - i was overseeing production programming languages at Google at the time, but getting to Typescript (which was the right choice) took a lot.
For most things, even things that seem to be contentious in the broader developer world, internal developer infrastructure teams were often relatively agnostic on choice as long as we had the resources to do it right (IE deal with migrations, etc). You'd have to push people to be meaningfully objective in evaluations, but once they realized you were not going to let them get away with nonsense, you got reasonable evaluations and options. Not always (can't avoid zealots at this scale), but a lot of the time.
But Typescript vs Dart vs Closure (GWT and a few other things were in there somewhere, too) was just particularly contentious for $reasons.
I know that Dart started as an alternative to JS, but it seems like JS target is now (unnecessarily) limiting Dart language in a way. I would be nice to be able to use lists of structs or a proper uint64 when needed. As a language it needs to expand in both directions to compete: high level productivity features and low level performance features. It has potential, but it's not there yet.
If Dart would find more use cases besides Flutter, it would make more sense to invest in its ecosystem.
It's pretty neat that Dart's JS support means you can take your code (e.g Flutter app) to the web, but I think that whole aspect of the ecosystem is underexplored/underdeveloped as of yet.
If you're within the Windows garden, those tools certainly make sense to use. But if you're not, there just simply isn't a reason to burden your app/platform with them.
To be clear, there's nothing wrong with C#, but the advocacy for it tends to be quite loud and passionate without much technical clarity in what it brings to the table that's lacking in other ecosystems. And again, you might be in for a world of hurt depending on how complex your needs are.
What exactly do you mean by this? How are syscalls worse in C# than other languages?
My understanding is other "cross-compiled" languages have cumbersome ergonomics with syscalls. They all use System or OS libraries that hide complexity and OS differences to varying degrees of success.
https://developers.redhat.com/blog/2016/09/14/pinvoke-in-net...
https://developers.redhat.com/blog/2019/03/25/using-net-pinv...
> without much technical clarity in what it brings to the table that's lacking in other ecosystems
What other GC language offers such levels of both high level expressiveness and low level control and also has a big ecosystem?
I would not, for example, advocate it for web UIs or any UIs except for Windows desktop UIs (and even there, I might advocate for JS based options).
>I would not, for example, advocate it for web UIs or any UIs except for Windows desktop UIs
Well, the GP was talking about using C# for Flutter, a cross-platform product from desktop to web lol.
Microsoft didn't want to compete with Sun so much as have an application development language with a garbage collector that wasn't owned by Sun.
You don't make much money off programming languages inherently.
This also elides an obvious riposte (so you mean they should have just used Mono? how did all that work out?) and a metric ton of differences between what C# targets and what Dart targets.
They still wanted a Java like ecosystem but they would be sure it only ran on Windows servers.
MS spent years being hostile to open source software. It's only in about the past decade that they've turned a corner.
Here's a famous email from Bill Gates about Java and how to stop it.
https://web.archive.org/web/20220630223035/https://www.teche...
But apparently lots of other people do run into them regularly, so I believe that such things do exist.
By the same token, I've heard countless reports of people struggling with the flexibility that Lisp offers, with co-workers who abuse it to create nightmarish situations. That you haven't experienced that doesn't mean no one does.
I don't mean "do dumb stuff", I mean I've literally never seen anyone redefine the `define` keyword in any code.
With javascript, I do see people use `===` frequently. It's a wart of the language that the operator even exist. It's not "dumb" to use it - it's how frequently are you assaulted with the bugs of the language (not bugs in your code).
You wouldn't let a child handle a chainsaw.
> I do think C# is by far the best mainstream language
My sense is that there are some legitimate reasons to pick something like Kotlin (JVM ecosystem), but a lot of folks that might have worked with C# in passing in the .NET Framework days simply haven't given the ecosystem another look. It's productive, stable, performant, and secure.
VS Code support is really good and Rider has a community license available.
That makes sense because C# can be much lower level and has its own set of priorities during compilation, so I'm not really complaining. But ergonomically, you really miss the TS type system when you don't have it.
There is also a greater selection of IDEs and LS's.
Please don't fan boy to the point of lying.
https://benhutchison.wordpress.com/2009/02/14/suns-rejection... https://stackoverflow.com/questions/1973579/why-doesnt-java-...
J++, which was Microsoft's Java implementation in the 90's added a few language extensions that were clearly not Sun-approved, but driven by internal engineering feedback at MS. C# having struct and class keywords, allowing you to define your own value types, is clearly a result of that missing in Java, which still in 2025 has no such equivalent yet.
Also Java's then native code interop solution, JNI, was and still remains complete garbage, and it's flaws were a huge guide for Microsoft when they deveoped .NET and it's native interop equivalent, PI (platform invoke).
Thankfully, Java now have FFM [foreign function and memory APIs](https://docs.oracle.com/en/java/javase/21/core/foreign-funct...) APIs (and also JNA which is community driven), which are much better than JNI.
You made 5 replies negative about C# in this comment section alone.
As they say, haters are fans too :)
Also why are you talking like cliquey high-school girl regarding a programming language. Complete with the emoji no less.
It's a tool, not a religion.