Golang vs. C# (.NET 5.0) at Benchmarks Game(benchmarksgame-team.pages.debian.net) |
Golang vs. C# (.NET 5.0) at Benchmarks Game(benchmarksgame-team.pages.debian.net) |
One of the standard caveats with this particular benchmark game with respect to Go is idiomatic optimizations are prohibited. To use the btree example, Go's memory management is low latency and non-moving, so allocations are expensive--any Go programmer writing a performance-sensitive btree implementation would pre-allocate the nodes in a single allocation--an absolutely idiomatic and trivial optimization--but the benchmark game requires that the nodes are allocated one at a time. In other words, the C# version is idiomatic, but the Go version is expressly contrived to be slower--not a very useful comparison.
Mad respect for .Net though; it's really impressive, I like the direction it's going, I'm glad it exists, etc.
Forcing allocations for every node isn't justified by a desire to demonstrate dynamically sized binary trees. A naive dynamically-sized tree would just keep a list of node buffers and allocate a new node buffer every time the previous one fills up (perhaps with subsequent buffers doubling in size). The benchmark is, by all appearances, contrived to be slower.
Which is not accepted for the C# programs either.
sync.Pool is accepted —
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
> the Go version is expressly contrived to be slower
The requirements were contrived in April 2008.
afaict Go initial release was March 2012.
Because C# doesn't benefit from this kind of optimization. Its GC is generational, which means that it has very fast allocations at the expense of high latency. In most applications, lower latency is more important than slower allocations (not least of all because these batch-allocating optimizations are nearly trivial), but these benchmarks don't reflect that at all.
> The requirements were contrived in April 2008. afaict Go initial release was March 2012.
Contrived = "the rules artificially prohibit idiomatic optimizations". It doesn't require that the maintainers have a prejudice against Go (although as you point out, the maintainers have had a decade to revisit their rules).
Overall you can see how fast Go is, it has little optimization compare to C# and it's as fast. Compare this: https://benchmarksgame-team.pages.debian.net/benchmarksgame/... and overly complicated C# version: https://benchmarksgame-team.pages.debian.net/benchmarksgame/... ( avx, Intrinsics etc ... )
It will intentionally use more memory for the sake of throughput, hence why this post has all .NET program flag for it, as it's a _speed_ benchmark primarily.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
I have to give you credit for trying to apply the Rule of Three to a single criticism.
Of course, I don't really understand how the fact that someone took the time to vectorize the C# submission is supposed to be a mark against C#...
There wasn’t even a lot of “modern C#” low level optimization like ref-structs/spans and similar. It actually looks like there is quite a bit of performance from C#8, 9, 10 left on the table.
Tons of very "interesting" attributes like this:
// prevent inlining into main to decrease JIT time to generate main
[SkipLocalsInit][MethodImpl(NoInlining)]
[SkipLocalsInit][StructLayout(LayoutKind.Explicit, Pack = 32)]
[SkipLocalsInit][MethodImpl(AggressiveOptimization | NoInlining)]
[FieldOffset(32)]
Not to mention that the entire file is using explicit vectorization, which I consider to be a very high degree of optimization -- tons of software never bothers to implement explicit vectorization, and does just fine.
If all of this is "nothing out of the ordinary", then "ordinary" C# has changed a lot since I last spent much time with it.
[0]: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Unfortunately developers who don’t know better judge it by it’s historical association with Windows rather than how powerful it is today.
I couldn’t find a direct C# to Rust comparison but Rust trying to compete with C++ means performance is a goal, if that’s what you are after.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Why does their web site have no contact nor link to where the source code for the project can be checked out, contributed to, or amended?
Perhaps they don't read the website text?
> … no contact nor link…
Search works.
At the very least, I couldn't realistically re-run some of the example benchmarks from the source embedded in the HTML, because they did not include vendoring/version information for external packages they depend on. That made me doubt the provenance of https://salsa.debian.org/benchmarksgame-team/benchmarksgame.
Some of us actually got to experience the entire journey from the old to new world first-hand. We started out as a .NET 3.5 Framework solution (windows only), and are now looking at a .NET 6 upgrade (any platform). Over the course of 7+ years, we went through all of the following frameworks:
3.5 => 4.0 => 4.5 => 4.6.2 => [netcore convert]
2.0 => 2.2 => 3.0 => 3.1 => 5.0 => ...
Some of the transitions were a little painful, but the same fundamental product survived the entire trip.
I don't know of many other development ecosystems where you can get away with something like this. If we didn't have the stability this ecosystem has to offer, we would not be in business today.
At the time I was hired, it was to modernize an Access application used internally, to try to sell it as a product.
.Net was still in beta at the time (this was early in 2001). I figured I might as well go with the flow and try it out.
It was a crazy ride, and I've since left the company ... but we went through every version from beta through 4.8 before I left.
I'm now using .Net 5 in Azure to power my new company's REST APIs.
Avoid vscode for C# development, unlike TS/JS (which is top of the line), the support for C# even in core is toy level.
I've always thought the whole ecosystem looked really productive, and the code I've had to review occasionally looked well structured and readable. But when I was starting out MSDN cost a fortune and I've never really considered trying to learn it.
Most applications I develop these days are web services. I write most things today in Go, but I used to work quite a bit with C#/Asp.NET applications.
Here's what I do to build a Go web application:
- go build .
Here's what I used to do with .NET:
- Open VS, wait about 30 seconds for it to finally start working
- Set the target, Rebuild All
- Publish to file, wait
- Eventually get a packaging error, predicated on some obscure tools dependency issue somewhere inside my 98KB .csproj file, which I have to fix by closing down VS, manually editing in Notepad, and re-opening VS
- Finally get a working build+publish, ok, let's take a look
- Oh, very nice, the publish directory weighs in at nearly a QUARTER GIGABYTE, and contains about 60 dll dependencies and a ton of entirely useless descriptor files.
- Well, okay, let's at least get this deployed to the test server to make sure it still plays nice with IIS, xcopy this over.
- Oh, IIS doesn't like this at all, now let's spend the next couple hours tracking down this insane .net framework dependency hell.
- Screw it, where's my whiskey?Seems pretty simple. Though the binary size is definitely not on par with go.
Can you elaborate on this? What does it mean that ".Net ships with Azure"?
But for many years, the per seat cost was higher, which scared many away
p44 "Oberon — The Overlooked Jewel" Michael Franz, in "The School of Niklaus Wirth".
https://www.google.com/books/edition/The_School_of_Niklaus_W...
It's an intentional choice, that's why it compiles code so fast. Also because of that it's a lot simpler than say GCC. Before Go, the Plan 9 C compiler was designed in a similar manner too (I think the Go compiler was forked from it).
I think the simplicity aspect is even more important than the compiling speed. It's easier and cleaner to keep the compiler simple and write optimized assembly code by hand when it's needed. That way, the compiler doesn't get so messy (fewer bugs, easier to maintain...) and the written program is of better quality (humans can produce better code than compilers).
I very much want my computer to work as hard as it can to make my code more performant for free. What I would like to see more is a separate debug and release build mode, where the former can go as fast as it can without optimizations, while the latter can be as slow as it wants and result in the most optimal binary it can produce. Zig does that for example.
They performed almost equivalently for RPS I believe.
In the space we work in, a vast majority of systems are written for either Java or .Net
(The scala mailing list archive doesn't seem to go back that far.)
No. They are allowed a library memory pool.
As-it-says: 'Please don't implement your own custom "arena" or "memory pool" or "free list" - they will not be accepted.'
For the particular case of n-body you could argue you are already in a pretty extreme HPC world and doing it without vectors would basically be a toy calculator. The problem then of course that C# isn’t really ever idiomatic for that. The question (as alaways) becomes about what to compare. Typical or pushed to the limit.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
I find this a common theme with Azure support and .NET
I spent a chunk of time recently (10s of hours) doing dumb C# vs Go benchmarks - files and networking, and nothing worth taking seriously - just, usually the part about being IO-bound was true. C# is really impressive and was just a little slower with the best async solutions I could come up with. The machinery for async has overhead, so do Go routines and channels … the first-pass, not very performant code was just a little faster and IMHO clearer with Go (but I’m much better with Go /shrug).
It basically just loads a JSON file into memory then allows you to query the data with 2 API endpoints.
C# does provide a memory pool implementation.
https://golang.org/doc/faq#garbage_collection
> They’re all allowed to use bespoke pools and custom allocators.
No. They are allowed a library memory pool.
As-it-says: 'Please don't implement your own custom "arena" or "memory pool" or "free list" - they will not be accepted.'
> … while Go’s allocator is slower…
So that tiny tiny program shows it's slower because it's slower.
Cart before horse — the binary trees are justified by a desire to demonstrate memory allocation.
2. Why use a binary tree benchmark in the first place if you’re going to limit the implementation to certain naive implementations (and again, only for one language)? Why not just measure allocations outright or at least call the benchmark “allocator performance”?
3. Showing allocation performance doesn’t help anyone understand the actual performance of the language, which is transparently what everyone uses these benchmarks for. If they wanted a general idea for language performance they would allow trivial, idiomatic optimizations. A benchmark that shows allocation performance is worthless, and a suite of benchmarks that includes a benchmark for allocation performance but not GC latency is worse than worthless: it’s misleading because latency is the more important concern and it’s what these bump allocators trade in order to get their fast allocation performance.
2. Again, not limited to only one language.
3. You are allowed an opinion.
It is mostly frontend HTML — the benchmarks game project deliverable is a (now static) website.
> I couldn't realistically re-run some of the example benchmarks from the source embedded in the HTML, because…
People have taken the simplest thing that will work approach: select/copy the program source code from the website, paste into a text editor and save; then build & run adapting the commands shown on every program page.
"Where can I get the program source code? — zip'd program source code"
line 11 ? in the README
> … vendoring/version information for external packages they depend on…
If the programs don't build/run with the latest GA external packages, they will be shown as "Make Error" "Bad Output" "Failed" until someone updates them.
A zipped source code archive inside a Git repository is very surprising (violation of principle of least astonishment). No wonder I couldn’t find the benchmark programs’ source; they aren’t even indexed for code search due to residing in a zip file.
To address that concern, there's now a link in the homepage banner.
Not sure what you're referring to here.
> No. They are allowed a library memory pool.
Yes, this is a contrived rule. In reality, a Go developer would write the extra ~dozen lines (all of the heavy lifting done by the builtin slice implementation) and call it a day.
> So that tiny tiny program shows it's slower because it's slower.
Tautology. It's slower because the contrived rules preclude idiomatic optimizations.
My point is that these contrived benchmarks don't indicate anything about the relative performance of these languages, but you keep responding with some variation of "but Go is slower in these benchmarks!" which everyone already agrees with. So unless you're going to actually address the point, I don't see the point in continuing on. It feels like you're hell-bent on using this thread for your personal programming language holy war, which is disinteresting to me (see again my first post) and against the site rules.
And a C# developer could write a program that would avoid GC, and a Java developer…, and…
It feels like you're hell-bent on using this thread for your personal programming language holy war…
Are those idiomatic? If so, then they should be permitted to apply those optimizations. Again, the whole point of benchmarks is to indicate real-world use.
> It feels like you're hell-bent on using this thread for your personal programming language holy war…
I've reiterated my substantial point over and over again ("contrived rules don't indicate real world use, which is the definitional purpose of benchmarks") and you still haven't addressed it. But in any case, perhaps if both of us think the other is waging a holy war, it's an indicator that the thread has run its course.
2. It is in practice, but regardless of the size of the cohort there’s no compelling reason for these limitations.
3. And you’re entitled to ignore reason. It cuts both ways.
2. Again, not limited to only one language. You are allowed an opinion about what is or is not compelling.
3. As before.
2. Doesn't matter whether it's exactly one language.
> You are allowed an opinion about what is or is not compelling.
It's not a matter of opinion. The definitional purpose of benchmarks is to indicate something about reality; if you contrive rules that cause the benchmarks to deviate from reality, they lose their utility as benchmarks. I've demonstrated that the rules are contrived (i.e., they prohibit real-world, idiomatic optimizations), so I think we can say as a matter of fact that these benchmarks aren't useful.
Of course, no one can force anyone else to see reason (but I don't have any interest in talking with unreasonable people).
2. You have repeatedly claimed "only for one language".
> I think we can say as a matter of fact…
Apparently that is your opinion.
> You have repeatedly claimed "only for one language".
How many languages are in practice prevented from using pre-allocation? How big is the cohort? Does it matter if it's exactly one or if it's two or three? Why are you fixating on this relatively irrelevant point rather than the more substantial point that has been reiterated a dozen times?
> Apparently that is your opinion.
In the same sense that "the sky is blue" is merely my opinion.
How many provide GC?
The substantial point is that you wish special treatment for Go lang.
> … "the sky is blue" is merely my opinion…
When all can see the vibrant orange red sunset.