strchr.com

https://www.strchr.com/ C and assembly language programming under Win32

Coincidentally, just before this was posted I read a similar analysis of Windows executables from 2008: https://www.strchr.com/x86_machine_code_statistics MOV is still the top, but it was followed by PUSH and CALL.

If you’re interested in diving deeper, the SMhasher benchmark suite tests a much more exhaustive list of hash functions (including wyhash) - https://github.com/rurban/smhasher https://www.strchr.com/hash_functions Is also a great write up of how some of the classic hashing algorithms perform of various data sets. https://nullprogram.com/blog/2018/07/31/ is an interesting approach on generating hashing functions automatically.

A great resource for hash function performance: https://www.strchr.com/hash_functions It includes a bunch of more modern hash functions the author missed.

For short strings, it basically doesn't help. For longer strings, it's a bit faster, it depends how bad what you were doing before was. Straight string comparison is not a great application for this in part because it's to intel's advantage to make string compare loops work fast without it. But you can be sure it speeds some customer's serious workload up significantly. That said, it all depends. It's great for "figure out if there is newline in the next 16 characters". But again, compared to basic SSE compares, it often falls down for random string comparison. See, e.g, https://www.strchr.com/strcmp_and_strlen_using_sse_4.2 which has timings on core i5. it's of course, likely possible to find processors where it is faster/slower.

Yes, speed for normal inputs is what's generally desired. A faster hash function can yield better overall performance than having fewer collisions. Here's some empirical evidence for this: https://www.strchr.com/hash_functions The speed is correlated only weakly with the number of collisions-- and using the modern x86 CRC32 instruction yields the best results.

> LLVM's libc++ Great find! So the library solutions are still often suboptimal, and it's even more easy to hide bad decisions in the C++ sources, so whoever has the approach "just use the default library" should be aware of that once the performance is important. Yes, even the simple multiplicative constants can significantly improve the hash if it by default doesn't do anything with the input! The libraries definitely should be fixed, and adding the multiplication step is really a simple and fast change for a great benefit. As an inspiration, Kernighan and Ritchie in their book about C used a simple number 31, and that simple hash is still quite good compared to much more complex and more recent solutions as K&R also haven't used the (I guess misleadingly named) "open addressing" for their hash table. Their solution is amazingly minimalistic and in that context amazingly good for chain hash tables. I wouldn't be surprised if just changing return __c; to return __c * 31; in the functions discovered would result in great improvement. The good side of such a constant is that it can give the fast and small code even on the old architectures where the "normal" multiplication is slow (e.g. even if there's no fast multiplier the result can be obtained by one shift and one subtraction!). Also on modern architectures using this constant can't result in any performance degradation but improving the hash behavior of these formerly unprocessed inputs guarantees…

AFAIK CRCs are not the fastest "hashes" you can get. Some non-cryptographic hashes outperform crcs by a large factor and provide longer checksums and much better statistical properties. E.g. see this: * https://www.strchr.com/hash_functions * https://jpountz.github.io/lz4-java/1.2.0/xxhash-benchmark/

Good homework is to implement an x86-64 mod/rm encoder. Quite a few corner cases. Sometimes you'll need to up-convert input parameters, because some combinations can't use 8-bit offset, for example, but require a 32-bit value. RSP can't use scaling, etc. There's also plenty of redundancy: https://www.strchr.com/machine_code_redundancy

I can't really answer that but your question reminded me of https://www.strchr.com/x86_machine_code_statistics Turns out that in practice a tiny number of instructions make up the vast majority of all used upcodes in a typical application (although it's not clear in this article how long the tail is, there could be hundreds of instruction in that "others" slice). My understanding is that there are many legacy instructions in x86 that are basically never used nowadays. They're still a problem for silicon designers because they still have to implement them for compatibility purposes but compiler developers can chose not to use them and instead select a more "risc-y" subset of instructions to do the bulk of the work.

There's no memory-to-memory XOR in x86. A XOR swap will need one register, the array address itself will take another, and the "single integer variable" one more. Then you also need the absolute value of the variable; there's another register. The single-expression abs that you specify needs at least two registers, one to hold the shifted value and one to hold the original one. Two alternatives which do not involve multiplication and are 3 instructions is here: https://www.strchr.com/optimized_abs_function

Writing an expression parser is probably the compiler "equivalent": http://www.strchr.com/expression_evaluator

It's possible that the difference is in the actual code: x86 machine code sometimes has multiple different encodings for the same assembler instruction. These would show up as single-byte differences in the binary. See http://www.strchr.com/machine_code_redundancy for some examples. Compilers might use this to stenographically hide additional data in the binary. Printer manufacturers already do something similar (https://en.wikipedia.org/wiki/Printer_steganography). I wouldn't be surprised if MS compilers embedded a hidden data in binaries -- it could be useful to track down malware authors; or identify software created with pirated MS tooling.

I found a clear explanation of the SIMD instructions here: http://www.strchr.com/strcmp_and_strlen_using_sse_4.2 . Apparently the instructions operate on strings up to 16 bytes in length.

I tried to compile such list in past, see http://www.strchr.com/links

CRC32 instructions: http://www.strchr.com/crc32_popcnt String processing instructions in SSE4.2: http://www.strchr.com/strcmp_and_strlen_using_sse_4.2 AES encryption instructions: http://en.wikipedia.org/wiki/AES_instruction_set So, if you didn't know about those, give yourself a point, because you nailed it. (No sarcasm.) There's a definite trend there.

also: http://www.strchr.com/strcmp_and_strlen_using_sse_4.2 SSE 4.2 introduces four instructions (PcmpEstrI, PcmpEstrM, PcmpIstrI, and PcmpIstrM) that can be used to speed up text processing code (including strcmp, memcmp, strstr, and strspn functions).

Ternary search tries attempt to solve this problem (among others): http://en.wikipedia.org/wiki/Ternary_search_tree http://www.strchr.com/ternary_dags

danluu knows much more about the subject matter than me. Meanwhile... If I take the main.cpp from http://www.strchr.com/media/crc32_popcnt.zip that he comparing and paste it into a new, default, VC2010 project. Then compile it as a Release build, then the mixed disassembly for the inner loop of POPCNT_HardwareSubbuN() looks like cnt += __popcnt(*(DWORD*)buf) + __popcnt(*(DWORD*)(buf + sizeof(DWORD) )) + __popcnt(*(DWORD*)(buf + sizeof(DWORD) * 2 )) + __popcnt(*(DWORD*)(buf + sizeof(DWORD) * 3 )); 00E71BE0 popcnt ebx,dword ptr [edx+8] 00E71BE5 popcnt esi,dword ptr [edx+0Ch] 00E71BEA add esi,ebx 00E71BEC popcnt ebx,dword ptr [edx+4] 00E71BF1 add esi,ebx 00E71BF3 popcnt ebx,dword ptr [edx] 00E71BF7 add eax,ebx 00E71BF9 add eax,esi buf += sizeof(DWORD) * 4; 00E71BFB add edx,10h 00E71BFE dec ecx 00E71BFF jne POPCNT_HardwareUnrolled+50h (0E71BE0h) which looks to me to be as good or better than his inline asm 00000001000013c3 popcntq %r10, %r10 00000001000013c8 addq %r10, %rcx 00000001000013cb popcntq %r11, %r11 00000001000013d0 addq %r11, %r9 00000001000013d3 popcntq %r14, %r14 00000001000013d8 addq %r14, %r8 00000001000013db popcntq %rbx, %rbx And does not have the odd 000000010000133e …

> MurmurHash3 wasn't _designed_ with low collision rates in mind There is a difference between "low collision rate" and "collision resistance". Real-world data often contain regular patterns that can cause "bad" hash functions to give very non-random output, e.g. numbers at fixed intervals, pointers all aligned to 4k boundaries, or long strings that differ only by a short suffix. For instance, early versions of Java would sample only a few of the characters in a long string to calculate the hash code, which caused horrendous performance if you stored a bunch of related file paths in a hash map. Most hash functions try to avoid bad behaviour on this kind of good-natured input, but some are better at it than others, due to careful design for random-like distribution, and low collision rates. http://www.strchr.com/hash_functions has examples of collision rates for different hashes, all non-cryptographic.

Oh jeez... I wish I could just assume you were trolling here. Virtual memory: This is used to give every process a flat and consistent address space. It is not just a synonym for 'swapping', and it will be around forever. String instructions: Seriously, are you just making it up as you go along? http://www.strchr.com/strcmp_and_strlen_using_sse_4.2 Custom opcodes: Usually done via API's that have an emulation or compatability function, DirectX, OpenGL for examples. LOCK existed: The article describes an atomic compare and write, not just a LOCK. In fact, MUTEX is implemented with the compare and exchange instruction in most OS's. http://marc.info/?t=113443157800005&r=1&w=2 I believe the no execute bit is a very recent addition to AMD64, and is not available at all on x86. Vector operations and such: Have you not heard of MMX, SSE or even graphics cards? Either I'm feeding a troll here, or you have no clues available to you.