The options mentioned above are guaranteed to generate big errors, so they are acceptable only when errors really do not matter, e.g. in graphics for games.
The only correct option for avoiding subnormals is to enable the underflow exception and handle it in a manner that is considered more suitable for that application than the use of subnormals.
Such options exist only for games or for experts in numeric analysis, not for general-purpose programming.
Obviously those lucky/unlucky enough to be writing 56k fixed precision code wouldn't have this concern, but other ones instead :)
I think flush to zero is probably the preferred strategy these days.
Fabian Giesen explains in more detail here: https://mastodon.gamedev.place/@rygorous/117277063419144390
Unfortunately, the Wikipedia article, while probably being accurate, doesn't give a clear and concise answer.
IE, is 0.0001 a subnormal? Or is it 0.000000000000000000001?
sign * 1.mantissa * 2 ^ exponent
However with this simple scheme the number zero and a relatively large gap around it cannot be represented (relatively large to the gap between the smallest and next smalles number that can be represented).
So there is a special case where for the smallest encodeable exponent the mantissa must also specify that 1. or 0. prefix. Because its a special case it needs special handling that clever silicon engineers might think is unimportant enough to handle in microcode instead of dedicated silicon.
x86 has a mode to assume that all such small numbers are actually equal to zero which can then be handle without microcode fallback. Technically its even a bit more complicated because x86 has two different float implementations and for at least SSE floats you can control the denormals-are-zero and flush-(denormals)-to-zero-(when writing) modes independently. GCC -ffast-math actual enables that mode for the entire main thread.
AFAIK ARM NEON always works in that mode so the Gravion and Apple benchmarks might be unfair here undless you compare with DAZ and FTZ enabled on Intel. No idea if the AMD benchmarks might have used different modes. Because the flags are global per thread you can easily have unrelated loaded libraries messing the benchmark up.
This was only true for ARMv7 NEON (32-bit). ARMv8 / AArch64 NEON is IEEE compliant.
Each number can be written in infinitely many ways, for example 12, 1.2E1, and 0.012E3 all are “twelve”
In (binary) IEEE floats, the canonical way to write floats is
significant × 2^exponent
Float operations normally return numbers satisfying that.
However, in IEEE, the exponent cannot be made arbitrary small. Because of that, some very small numbers cannot be represented that way.
In those cases the standard says operations can return numbers with the value closest to the correct value with a significant less than 1. Those number representations are called subnormals.
Does that help you?
[Edited: correct decimal after noticing that my calculator defaulted to the wrong setting]
[And again because I think there's a bug in the last few digits, so debugging that's a fun activity for the weekend]
[And a third time because nope, those were correct and I can't type]
Mechanically speaking, the two zeros use the subnormal number format, so in that sense they are subnormal (but definitionally they aren't). Also, I guess FPUs treat zero differently from other subnormal numbers, which is why zero doesn't have a performance penalty.
Relevant articles to read: https://stackoverflow.com/questions/73890260/why-is-zero-not... , https://en.wikipedia.org/wiki/Sterbenz_lemma , https://en.wikipedia.org/wiki/Subnormal_number
(And thankfully, for when I work with small numbers, they are still much larger than that.)
This problem is fixed by reserving one of the exponents for the representation of 0. Some of the formats (e.g. VAX floating point) that introduced this implicit-1-bit for the binary format said that every number with this special-0-exponent was a zero. But IEEE 754 introduced the concept of gradual underflow, and says instead that it is a bit string with the implicit digit before the decimal point as a 0 instead of 1.
Putting it differently and more succinctly: a subnormal number is a number that has fewer digits of precision than is normally implied by the format. Which numbers are subnormal numbers is entirely dependent on the floating-point format.
between any two numbers, there's basically the same epsilon difference, but from the smallest number to zero it's bigger.
A subnormal number breaks that convention, it just becomes 0.bbbbb... * 2^-N. As the numbers get smaller, the relative difference between the numbers gets larger. That also means their precision is smaller than the normal floats.
Most "mundane" uses of floating point have no need for subnormal numbers, and results that underflow could just be flushed to zero. But they’re sometimes important in scientific computing to ensure sufficient smoothness around zero, avoiding precision issues.
This was very good, because underflows completely break the assumptions about floating-point arithmetic on which numeric algorithms are based, so the errors in the final results become unpredictable.
Subnormal numbers have been introduced as a means to avoid handling every underflow exception, because typically the use of subnormals eliminates the errors that would otherwise be caused by underflows.
The flush-to-zero and denormals-of-zero options must be strictly forbidden for any general-purpose applications. They should be permitted only in applications where there is no doubt that regardless how big the errors will be they will not have any really harmful effect, which is true for games and perhaps for AI, but for little else.
This is another great misfeature promoted by Intel, in order to win meaningless benchmarks. It would have been much better if these standard-breaking features would not have existed, because they are much more often used when they should not be used, than when they are harmless.
More generally, subnormals are needed for Sterbenz Lemma to hold everywhere: https://en.wikipedia.org/wiki/Sterbenz_lemma
Have a look at https://en.wikipedia.org/wiki/Subnormal_number for some context.
I haven't done any work on this stuff since 2019, so my memory may be hazy.
When telephony transitioned from analog voice transmission to digital, the PCM (pulse-code modulation) encoded audio signal used 8-bit samples, which were a form of 8-bit floating-point numbers (with American and European encoding variants: mu-law and A-law). The use of a floating-point format enabled the 8-bit samples to have a dynamic range as big as for a 12-bit or 13-bit fixed-point encoding.
Subnormals where used in digital telephony, because otherwise the errors around zero would have been so great that the voice audio would not have been intelligible.
In general, in smaller floating-point formats the use of subnormals is even more important than in bigger formats, in order to avoid the loss of precision around zero.
16-bit: https://en.wikipedia.org/wiki/Half-precision_floating-point_...
If the use of subnormals is disabled with FTZ/DAZ that is guaranteed to generate big errors and it is completely unpredictable how big the errors will be.
If a computational algorithm generates underflows at some place, there is no way to modify the algorithm so that flushing-to-zero will not make any difference (i.e. no errors).
What is possible, is to modify the algorithm so that underflows will never happen.
This was the traditional way of writing numeric algorithms. Because on early computers underflows would crash the program, the same as overflows, one had to improve the algorithm in order to avoid both underflows and overflows.
Subnormals and infinities have been introduced in the standard precisely for lazier programmers, so that they would be able to avoid the rewriting of algorithms without the risks that underflows and overflows would generate major errors.
Unfortunately, it seems that for some programmers this is still not enough, because they want simultaneously to not be bothered with rewriting the algorithms and to have the program run as fast as with an optimized algorithm.
For this, the solution is very simple and it is not enabling FTZ/DAZ, which unless is done for a game might cause unpredictable financial losses for an unsuspecting customer, who expects that a computer must provide correct results.
The right solution is to not buy Intel CPUs or any other kind of processors whose vendor believes that the correctness of computations does not matter. It should be noted however, that the Intel server CPUs use CPU cores that are obsolete in desktop and laptop CPUs, i.e. the tested Intel CPUs use cores like those in Meteor Lake and Raptor Lake CPUs. I do not know if the more recent Intel CPU cores, from Panther Lake/Arrow Lake S/Arrow Lake H/Lunar Lake, have retained this Intel misfeature, which has characterized the Intel CPUs for much more than a decade.
If someone says that they have enabled FTZ/DAZ and they did not see any significant difference in the results of a program, that is complete B*S*T, because it is impossible to test exhaustively any program that does floating-point computations and the errors are expected to happen only for certain values, which are unlikely to be encountered during testing, but you cannot predict that those values will not be encountered in production.
Some poster has linked a Mastodon thread, where Fabian Giesen explains that handling in hardware the subnormals is cheap in floating-point adders and in fused-multiply-add (FMA) execution units. Many processors do the multiplications in the FMA execution units, so there is no penalty for them to do the subnormal handling in the right way.
On the other hand, some CPUs, including the Intel big cores, have some floating-point multipliers that are separate from the FMA units. The reason is that those separate multipliers can have lower latencies, typically by 1 or 2 clock cycles, which may help those CPUs to win some benchmarks, especially when running unoptimized legacy programs (in optimized programs, most multiplications are combined with additions into FMA operations).
The separate multipliers are simplified in comparison with those included in the FMA units, and handling subnormals in them would be expensive, because then they would become so complex that there would be no advantage for them to be separate multipliers. Which is why Intel does not handle subnormal multiplication in hardware, but a microprogram is invoked for this.
A single sequence of floating-point operations executed on an IEEE 754 compliant CPU has always been perfectly deterministic since the first version of the standard.
Who wants a deterministic computation must use a single-threaded program or a multi-threaded program where the order of execution is deterministic, and one must be careful so that changes in compiler versions or compilation options will not affect the type and order of the operations that are executed.