https://www.igorslab.de/en/intel-deactivated-avx-512-on-alde...
AVX-512 was never really supported in newer consumer CPUs with heterogeneous architecture. These CPUs have a mix of powerful cores and efficiency cores. The AVX-512 instructions were never added to the efficiency cores because it would use way too much die space and defeat the purpose of efficiency cores.
There was previously a hidden option to disable the efficiency cores and enable AVX-512 on the remaining power cores, but the number of workloads that would warrant turning off a lot of your cores to speed up AVX-512 calculations is virtually non-existent in the consumer world (where these cheap CPUs are targeted).
The whole journalism controversy around AVX-512 has been a bit of a joke because many of the same journalists tried to generate controversy when AVX-512 was first introduced and they realized that AVX-512 code would reduce the CPU clock speed. There were numerous articles about turning off AVX-512 on previous generation CPUs to avoid this downclocking and to make overclocks more stable.
And this is why scalable vector ISA's like the RISC-V vector extensions are superior to fixed-size SIMD. You can support both kinds of microarchitecture while running the exact same code.
Isn't the purpose of efficiency cores to be more power efficient? It's more power efficient to vectorize instructions and minimize pipeline re-ordering.
I just got through doing some work with vectorization.
On the simplest workload I have, splitting a 3 MByte text file into lines, writing a pointer to each string to an array, GCC will not vectorize the naive loop, though ICC might I guess.
With simple vectorization to AVX512 (64 unsigned chars in a vector), finding all the line breaks goes from 1.3 msec to 0.1 msec, so a little better than a 10x speedup, still just on the one core, which keeps things simple.
I was using Agner Fog's VCL 2, Apache licensed C++ Vector Class Library. It's super easy.
Still, what does it signal that vector extensions are required to get better string performance on x86? Wouldn't it be better if Intel invested their AVX transistor budget into simply making existing REPB prefixes a lot faster?
NN-512 is cool. I think the Go code is pretty ugly but I like the concept of the compiler a lot.
My question is whether Intel investing in AVX-512 is wise, given that: -Most existing code is not aware of AVX anyway; -Developers are especially wary of AVX-512, since they expect it to be discontinued soon.
Consequently, wouldn't Intel be better off by using the silicon dedicated to AVX-512 to speed up instruction patterns that are actually used?
AVX is just the SIMD unit. I would argue the transistors were spent on SIMD, and the hitch is simply the best way to send str commands to the SIMD hardware.
I don't have an AVX512 machine with VBMI2, but here's what my untested code might look like:
__m512i spaces = _mm512_set1_epi8(' ');
size_t i = 0;
for (; i + (64 * 4 - 1) < howmany; i += 64 * 4) {
// 4 input regs, 4 output regs, you can actually do up to 8 because there are 8 mask registers
__m512i in0 = _mm512_loadu_si512(bytes + i);
__m512i in1 = _mm512_loadu_si512(bytes + i + 64);
__m512i in2 = _mm512_loadu_si512(bytes + i + 128);
__m512i in3 = _mm512_loadu_si512(bytes + i + 192);
__mmask64 mask0 = _mm512_cmpgt_epi8_mask (in0, spaces);
__mmask64 mask1 = _mm512_cmpgt_epi8_mask (in1, spaces);
__mmask64 mask2 = _mm512_cmpgt_epi8_mask (in2, spaces);
__mmask64 mask3 = _mm512_cmpgt_epi8_mask (in3, spaces);
auto reg0 = _mm512_maskz_compress_epi8 (mask0, x);
auto reg1 = _mm512_maskz_compress_epi8 (mask1, x);
auto reg2 = _mm512_maskz_compress_epi8 (mask2, x);
auto reg3 = _mm512_maskz_compress_epi8 (mask3, x);
_mm512_storeu_si512(bytes + pos, reg0);
pos += _popcnt64(mask0);
_mm512_storeu_si512(bytes + pos, reg1);
pos += _popcnt64(mask1);
_mm512_storeu_si512(bytes + pos, reg2);
pos += _popcnt64(mask2);
_mm512_storeu_si512(bytes + pos, reg3);
pos += _popcnt64(mask3);
}
// old code can go here, since it handles a smaller size well
I found myself wondering if one could create a domain-specific language for specifying string processing tasks, and then automate some of the tricks with a compiler (possibly with human-specified optimization annotations). Halide did this sort of thing for image processing (and ML via TVM to some extent) and it was a pretty significant success.
https://ark.intel.com/content/www/us/en/ark/search/featurefi...
The author mentions it's difficult to identify which features are supported on which processor, but ark.intel.com has a quite good catalog.
It is true that randomly applied AVX-512 instructions can cause a slight clock speed reduction, the proper way to use libraries like this would be within specific hot code loops where the mild clock speed reduction is more than offset by the huge parallelism increase.
This doesn’t make sense if you’re a consumer doing something multitasking and a background process is invoking the AVX-512 penalty in the background, but it usually would make sense in a server scenario.
Forget about 512 bit vectors or FMAs.
The more generous interpretation is that Intel fixed that issue a while back although the CPUs with that problem are still in rotation and you have to think about that when compiling your code.
The GPU is incredible at raw throughput, and this particular problem can actually implemented fairly straightforwardly (it's a stream compaction, which in turn can be expressed in terms of prefix sum). However, where the GPU absolutely falls down is when you want to interleave CPU and GPU computations. To give round numbers, the roundtrip latency is on the order of 100µs, and even aside from that, the memcpy back and forth between host and device memory might actually be slower than just solving the problem on the CPU. So you only win when the strings are very large, again using round numbers about a megabyte.
Things change if you are able to pipeline a lot of useful computation on the GPU. This is an area of active research (including my own). Aaron Hsu has been doing groundbreaking work implementing an entire compiler on the GPU, and there's more recent work[1], implemented in Futhark, that suggests that that this approach is promising.
I have a paper in the pipeline that includes an extraordinarily high performance (~12G elements/s) GPU implementation of the parentheses matching problem, which is the heart of parsing. If anyone would like to review a draft and provide comments, please add a comment to the GitHub issue[2] I'm using to track this. It's due very soon and I'm on a tight timeline to get all the measurements done, so actionable suggestions on how to improve the text would be most welcome.
[1]: https://theses.liacs.nl/pdf/2020-2021-VoetterRobin.pdf
[2]: https://github.com/raphlinus/raphlinus.github.io/issues/66#i...
I can't help but notice that, at least in my experience on Windows, this is the same order of magnitude as for inter-process communication on the local machine. Tangent: That latency was my nemesis as a Windows screen reader developer; the platform accessibility APIs weren't well designed to take it into account. Windows 11 finally has a good solution for this problem (yes, I helped implement that while I was at Microsoft).
I personally believe it may be possible to reduce latency using techniques similar to io_uring, but it may not be simple. Likely a major reason for the roundtrips is so that a trusted process (part of the GPU driver) can validate inputs from untrusted user code before it's presented to the GPU hardware.
The original name of AVX-512 was "Larrabee New Instructions". Unlike with the other Intel instruction set extensions, the team which defined the "Larrabee New Instructions" included graphics experts hired from outside Intel, which is probably the reason why AVX-512 is a better SIMD instruction set than all the other designed by Intel.
Unfortunately, Sandy Bridge (2011), instead of implementing a scaled-down version of the "Larrabee New Instructions", implemented the significantly worse AVX instruction set.
A couple of years later, Intel Haswell (2013), added to AVX a few of the extra instructions of the "Larrabee New Instructions", e.g. fused multiply-add and memory gather instructions. The Haswell AVX2 was thus a great improvement over the Sandy Bridge AVX, but it remained far from having all the features that had already existed in LRBni (made public in 2009).
After the Intel Larrabee project flopped, LRBni passed through a few name changes, until 2016, when it was renamed to AVX-512 after a small change in the binary encoding of the instructions.
I also dislike the name "AVX-512", but my reason is different. "AVX-512" is made to sound like it is an evolution of AVX, while the truth is the other way around, AVX was an involution of LRBni, whose purpose was to maximize the profits of Intel by minimizing the CPU manufacturing costs, taking advantage of the fact that the competition was weak, so the buyers had to be content with the crippled Intel CPUs with AVX, because nobody offered anything better.
The existence of AVX has caused a lot of additional work for many programmers, who had to write programs much more complex than it would have been possible with LRBni, which had from the beginning features designed to allow simplified programming, e.g. the mask registers that allow much simpler prologues and epilogues for loops and both gather loads and scatter stores for accessing the memory.
So it's more like both groups knew what the other was doing, but LRBni was free to focus primarily on graphics and a clean slate, while the AVX folks shot for "SSE but wider, and a few more".
AVX-512 is sort of a franken-combo of what AVX3 would have been, plus many of the LRBni instructions that shipped in the poorly named MIC parts, plus some more (e.g., now including a VNNI dialect, bf16 ops, etc.).
Most of the development of LRBni was between 2005 and 2009, when it became publicly known. The first product with LRBni was Knights Ferry, which was introduced in 2010, being made with the older 45-nm process. Knights Ferry was used only in development systems, due to insufficient performance.
Sandy Bridge, using the newer 32-nm process, was launched in 2011. I do not know when the development of Sandy Bridge had started, but in any case the first few years of development must have overlapped with the last few years of the development of LRBni.
I suppose that there was little, if any, communication between the 2 Intel teams.
AVX was developed as an instruction set extension in the same way as the majority of the instruction set extensions had been developed by Intel since the days of Intel 8008 (1972) and until the present x86 ISA.
Intel has only very seldom introduced new instructions that had been designed having a global view of the instruction set and making a thorough analysis of which instructions should exist in order to reach either the best performance or the least programming effort.
In most cases the new instructions have been chosen so that they would need only minimal hardware changes from the previous CPU generation for their implementation, while still providing a measurable improvement in some benchmark. The most innovative additions to the Intel ISA had usually been included in the instruction sets of other CPUs many years before, but Intel has delayed to also add them as much as possible.
This strategy of Intel is indeed the best for ensuring the largest profits from making CPUs, as long as there is no strong competition.
Moreover, now the quality of the ISA matters much less for performance than earlier, because the very complex CPUs from today can perform a lot of transformations on the instruction stream, like splitting / reordering / fusion, which can remove performance bottlenecks due to poor instruction encoding.
Most programmers use only high-level languages, so only the compiler writers and those that have to write extremely optimized programs have to deal with various ugly parts of the Intel-AMD ISA.
So AVX for Sandy Bridge has been designed in the typical Intel way, having as target to be a minimal improvement over SSE.
On the other hand LRBni was designed from the ground, to be the best instruction set that they knew how to implement for performing its tasks.
So it was normal that the end results were different.
For the Intel customers, it would have been much better if Intel did not have 2 divergent developments for their future SIMD ISA, but they would have established a single, coherent, roadmap for SIMD ISA development during the next generations of Intel CPUs.
In an ideal company such a roadmap should have been established after discussions with a wide participation, from all the relevant Intel teams.
For cost reasons, it is obvious that it would not have been good for Sandy Bridge to implement the full LRBni ISA. Nevertheless, it would have been very easy to implement a LRBni subset better than AVX.
Sandy Bridge should still have implemented only 256-bit operations, and the implementation of some operations, e.g. gather and scatter, could have been delayed for a later CPU generation.
However other LRBni features, should have been present since the beginning, e.g. the mask registers, because they influence the instruction encoding formats.
The mask registers would have required very little additional hardware resources (the actual hardware registers can reuse the 8087 registers), but they would have simplified AVX programming a lot, by removing the complicated code needed to handle correctly different data sizes and alignments.
The current CPUs with AVX-512 support would have been simpler, by not having to decode 2 completely distinct binary instruction formats, for AVX and for AVX-512, which is a fact that made difficult the implementation of AVX-512 in the small cores of Alder Lake.
Actually, I stand corrected, after double checking, Cloudflare were using Silver. Entry level data centre chips, instead of small business chips. Still not the kind of chips you'd buy for high performance infrastructure, and not intended to be used for such.
Xeon Silver 4116s hit the market at $1,002.00. The Golds were $1,221.00. The performance differences are quite significant. For something that'll be in service for ~3-5 years, $200 is absolutely trivial by way of a per-chip increase. It's firmly in the "false economy" territory to be skimping on your chip costs. It's a bit more understandable in smaller businesses, but you just don't do it when you're operating at scale.
Also remember: at the scales that Cloudflare are purchasing at, they won't be paying RRP. They'll be getting tidy discounts.
Anyway I’m sure they compared the TCO of buying more low-end chips vs fewer high-end chips.
If you disable E cores you could enable AVX-512 on certain motherboards, but like I said that’s not really a net win 99.99% of the time when you’re giving up entire cores.
It was also at your own risk because presumably the power/clock speed profiles were never tuned for a feature that wasn’t actually supported. I can see exactly why they turned it off on newer CPUs only after an announcement.
Only because they screwed it up on purpose! That's not an acceptable reason for removing the feature; in part because it would apply to any feature they decided to cut.
Writing code for a specific SIMD instruction set is non-trivial, but most code will get some benefit by being compiled for the right ISA. You don't get the really fancy instructions because the pattern matching in the compiler isn't very intelligent but quite a lot of stuff is going to benefit by magic.
Even without cutting people without some AVX off, you can have a fast/slow path fairly easily.
Sure, REP STOSB/MOVSB make for a very compact memset/memcpy, but their performance varies depending on CPU feature flags, so you're going to want multiple codepaths anyway. And vector instructions are vastly more flexible than just those two.
Also, I have not met developers who expect AVX-512 to be discontinued (the regrettable ADL situation notwithstanding; that's not a server CPU). AMD is actually adding AVX-512.
For which percentage of users?
> AMD is actually adding AVX-512
Which is irrelevant to in-market support for that instruction set.
Anyone using software that benefits from vector instructions. That includes a variety of compression, search, and image processing algorithms. Your JPEG decompression library might be using SSE2 or Neon. All high-end processors have included some form of vector instruction for like 20+ years now. Even the processor in my old eBook reader has the ARM Neon instructions.
Personally I don't find the e-cores on my alder lake CPU to be of any value. They're more of a hazard than a benefit.
If anything I'd say that Core 59 is one of those exceptions that prove the rule. This is such a rare phenomenon that when it does happen you can do the work to pin it down and say yup, this CPU is busted - if it was really commonplace you'd constantly trip over these bugs and get nowhere. There probably isn't really, as that paper claims, a "systemic issue across generations" except that those generations are all running Facebook's buggy code.
https://man7.org/linux/man-pages/man2/sched_setaffinity.2.ht...