Simon Byrne

"Friends don't let friends use fast-math" https://simonbyrne.github.io/notes/fastmath/

Did you mean -fbroken-and-not-necessarily-fast-math? [1] [1] But really, if -ffast-math does turn -funsafe-math-optimizations on, it should have been named similarly. There is a possibility of much safer -ffast-math with almost zero breakage (by assuming a subset of IEEE 754, like the fixed rounding mode). The current -ffast-math is so reckless [2]. [2] https://simonbyrne.github.io/notes/fastmath/#flushing_subnormals_to_ze…

See also https://simonbyrne.github.io/notes/fastmath/ for a similar story in julia, where ffast-math is now banned for C/C++/Fortran dependencies

Thanks :) I'd be interested to hear how it goes for you. Agree that 4x unrolling is getting most of the low-hanging fruit without excessive code size. I saw only very slightly better performance on SKX with 8x. You're right that it's nicer when the compiler can decide about the unrolling - for example with knowledge whether we have 16 or 32 regs. The unsafe/fast-math flags are pretty dangerous, though :/ https://simonbyrne.github.io/notes/fastmath/ Especially when they enable flush-to-zero, which would be unacceptable for a library loaded into some other application.

https://simonbyrne.github.io/notes/fastmath/ also has a nice discussion about the possible pitfalls of "fast"-math.

No, because as I commented in another subthread, `-ffast-math` is: 1. dangerous assumptions hidden behind a simple, attractive-looking option [1]. It should be called -fwrong-math or -fdangerous-math or something (GCC does have the funnily named switch -funsafe-math-optimizations – what could go wrong with fun, safe math optimizations?!) 2. Translation-unit scoped, which means that dependencies not consented to "fast math" can break your code (as in UB land) or make the optimizations pointless, and your code can break your dependencies' semantics too via inlining. On the other hand, a library author must think very carefully what float opts to enable in order to be compatible with client code. Deciding how the scoping of non-IEEE float math operations should work is a very nontrivial question. The scope could be a translation unit, a module, a type, a function, a block, or every individual operation, and none of those is without issues, particularly regarding questions like inlining and interprocedural and link-time-optimization, as well as ergonomics. In other ways, it's yet another function coloring problem. There are currently-unstable "algebraic_add/mul/etc" methods for floats for letting LLVM treat those particular operations as if floats were real numbers [2]. They're the first step towards safe UB-free float optimizations, but of course those names are rather awkward to use in math-heavy code, and a wrapper type overloading the normal operators would be good to…

"I'm never sure what -ffast-math does exactly." Me too when I am away from C for a while. The topic has been on HN [3] * Enable the use of SIMD instructions * alter the behavior regarding NaN (you can't even check for NaN afterwards with isnan(f)) * alter the associativity of expression a+(b+c) might become (a+b)+c which seems inconspicuous at first, but there are exceptions (as example see [1] under -fassociative-math) * change subnormals to zero (even if your program isn't compiled with this option, but a library you link to your program). A nice overview from which I summarize is in [1] which contains a link to [2] with this nice text: "If a sufficiently advanced compiler is indistinguishable from an adversary, then giving the compiler access to -ffast-math is gifting that enemy nukes. That doesn’t mean you can’t use it! You just have to test enough to gain confidence that no bombs go off with your compiler on your system" [1] https://simonbyrne.github.io/notes/fastmath/ [2] https://discourse.julialang.org/t/when-if-a-b-x-1-a-b-divides-by-zero/… [3] https://news.ycombinator.com/item?id=29201473 (107)

fastmath is bad in general. In Julia it's not as bad as many other languages because it's attempted to be kept local. The @fastmath macro is essentially a find-replace macro, where it finds things like ^ and replaces it with the fast_pow function which drops the 1ulp requirement. However, this global option was really the one piece left in Julia that where it could creep in, hence the reason to drop it. There are still some other difficulties of course, since fastmath in the C ABI is quite wild (or I guess, it's really the GCC implementation up to GCC 13 (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55522#c45)). Simon wrote a nice piece about the difficulties in general: https://simonbyrne.github.io/notes/fastmath/. In a general sense there is still the potential vulnerability that effects the Python ecosystem which is that if any package has binaries built with fastmath it could cause other calculations to be fastmath as well in a non-local way (https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subno… > It turns out (somewhat insanely) that when -ffast-math is enabled, the compiler will link in a constructor that sets the FTZ/DAZ flags whenever the library is loaded — even on shared libraries, which means that any application that loads that library will have its floating point behavior changed for the whole process. And -Ofast, which sounds appealingly like a "make my program go fast" flag, automatically enables -ffast-math, so some projects may…

>In particular all basic operations should produce bit-exact results to last ulp. As long as you are not using a compiler that utilizes x87's extended precision flaots for intermediate calculations, and silently rounding whenever it transfers to memory (That used to be a common issue), and as long as you are not doing dumb stuff with compiler math flags. Also if you have any code anwhere in your program that relies on correct subnormal handling, then you need to be absolutely sure no code is compiled with `-ffast-math`, including in any dynamically loaded code in your entire program, or your math will break: https://simonbyrne.github.io/notes/fastmath/#flushing_subnormals_to_ze… And of course if you are doing anything complicated with floating point number, there are entire fields of study about creating numerically stable algorithms, and determining the precision of algorithms with floating point numbers.