Simon Sapin

I’ve been sharing it multiple times but I love it: WTF-16 spec https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Yep. Microsoft is the main reason for 2 of my favourite unicode links. https://utf8everywhere.org/ and https://simonsapin.github.io/wtf-8/ And, apparently it's mildly inaccurate to say it uses UTF-16... it's more like UCS-2 with UTF-16 hacked in, with no validation. Thus WTF-8.

Surrogates are technically a UTF-16 only thing. Realizing that sometimes they nevertheless escape out into the wild, WTF-8 defines a superset of UTF-8 that encodes them: https://simonsapin.github.io/wtf-8/ To be clear, this is not an official Unicode spec. It's a hack (albeit a pretty natural and obvious one) to deal with systems that don't do Unicode quite right. I recently came across some old code that narrows wchar_t to UCS-2 by zeroing out the high-order bytes. Even though my test was careful not to generate any surrogates in the input, they showed up in the output when a randomly generated code point like U+1DF7C was mangled into U+DF7C. A corrupted value like that is not necessarily a great example of something you want to preserve, but it's the sort of thing that late 90s code assumed about Unicode.

My favorite bit of learning with Rust strings was the WTF-8 format: https://simonsapin.github.io/wtf-8/ Implemented: https://github.com/rust-lang/rust/blob/master/src/libstd/sys/common/wt…

For the benefit of others (the link is nonobvious), here's Markus Kuhn's presentation of UTF-8B: http://hyperreal.org/~est/utf-8b/releases/utf-8b-20060413043934/kuhn-u… The tl;dr is to map an invalid UTF-8 byte n to code point U+DC00 + n, which puts it in the code point range reserved for the second part of a surrogate pair. (In UTF-16, a 16-bit value between D800 and DBFF followed by a 16-bit value between DC00 and DFFF is used to encode a code point that cannot fit in 16 bits. Since these "surrogate pairs" happen only in that order, there is room to extend UTF-16 by assigning a meaning to a DC00-DFFF value seen without a D800-DBFF before it.) Since the surrogate code points are defined as not "Unicode scalar values" and cannot exist in well-formed "Unicode text", and therefore cannot be decoded from well-formed UTF-8, there's no risk of confusion. There are some similarities with the extension of UTF-8 encoding that is sometimes called "WTF-8" https://simonsapin.github.io/wtf-8/. WTF-8 lets unchecked purportedly-UTF-16 data be parsed as a sequence of code points, encoded into an extension of UTF-8, and round-tripped back into the original array of uint16s. UTF-8B lets unchecked purportedly-UTF-8 data be parsed as a sequence of code points, encoded into an extension of UTF-16, and round-tripped back into the original array of uint8s. They're not quite compatible, because WTF-8 would encode U+DC80 as a three-byte sequence (ED B2 80), and UTF-8B would decode that…

No, WTF-8[1] is a precisely defined format (that isn't that). If you imagine a format that can encode JavaScript strings containing unpaired surrogates, that's WTF-8. (Well-formed WTF-8 is the same type as a JS string, through with a different representation.) (Though that would have been cute name for the UTF-8/latin1/UTF-8 fail.) [1]: https://simonsapin.github.io/wtf-8/

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

WTF-8 is actually a real encoding, used for encoding invalid UTF-16 unpaired surrogates for UTF-8 systems: https://simonsapin.github.io/wtf-8/

UCS-2 is why we have the WTF-8 encoding standard, which allows mismatched UTF-16 surrogate pairs to survive a round-trip through an 8-bit encoding. https://simonsapin.github.io/wtf-8/

That's not what invalid UTF-8 means. It doesn't mean replacing codepoints with other codepoints, it means that the encoding of the codepoints is invalid. Since JavaScript uses UTF-16 strings, it's possible to create JavaScript strings that don't have a valid UTF-8 representation (I believe unpaired surrogates can do this). Also, if you are working with raw bytes, there is no guarantee that it has any encoding at all (this seems to be what the article is about). There is WTF-8 [0] that proposes a possible solution to the UTF-16 problem, but purely as an internal representation and not as an interchange format. I think the Rust standard library uses WTF-8 to represent Windows wide-strings, which allows ASCII and UTF-8 algorithms to work on them, while still being able to represent any UTF-16 string. [0]: https://simonsapin.github.io/wtf-8/

Rust strings enforce utf-8 encoding, yes. However, it seems Windows (which uses utf-16) allows ill-formed UTF-16, in particular it tolerates unpaired surrogates. You can read more here http://simonsapin.github.io/wtf-8/

Thanks for sharing! This is the first time I saw a mention of WTF-8 [1] and WTF-16. From the spec description this seems strange to use this interoperability "hack" (using the word from the spec) as the foundation of the string proposal. I wonder if they could use UTF-8 instead and keep WTF-16 for interoperability with JavaScript. > WTF-8 [...] is a superset of UTF-8 that encodes surrogate code points if they are not in a pair. It represents, in a way compatible with UTF-8, text from systems such as JavaScript and Windows that use UTF-16 internally but don’t enforce the well-formedness invariant that surrogates must be paired. WTF-8 is a hack intended to be used internally in self-contained systems with components that need to support potentially ill-formed UTF-16 for legacy reasons. [1] https://simonsapin.github.io/wtf-8/

I think NTFS is UCS-2, that is why WTF-8 was invented https://simonsapin.github.io/wtf-8/

Because that's how UTF-8 is defined[1]. WTF-8 lifts that restriction. [1] https://simonsapin.github.io/wtf-8/#utf-8

Windows is still using WTF-16 for filenames, right? https://simonsapin.github.io/wtf-8/#wtf-16

The value spaces are in fact asymmetric. In Unicode jargon, UTF-8 is a "list of Unicode scalar values" while WTF-16, i.e. UTF-16 as seen in practice, is a "list of Unicode code points (except surrogate pairs)". Unicode code points are a superset of Unicode scalar values, aka "list of Unicode Code Points except surrogate code points", hence conversion from WTF-16 to UTF-8 is lossy. In WTF-16 languages, this is not a problem because the system is designed for WTF-16, but it becomes one if the value space is restricted further, here to UTF-8. That's why, in such mixed systems, https://simonsapin.github.io/wtf-8/ is typically used. Not UTF-8. And Wasm is such a mixed system.

Yeah, there's sometimes a lot more hacks like WTF-8 and WTF-16 in practice on UCS-2 originally systems (including Windows and JS) than is healthy: https://simonsapin.github.io/wtf-8/

Is CESU-8 a synonym of WTF-8? https://en.wikipedia.org/wiki/UTF-8#WTF-8 ; https://simonsapin.github.io/wtf-8/

This is what I would like to see as a logical conclusion that is future proof: The query and response MUST be transmitted and stored 'as is' (a sequence of octets). The query and response SHOULD be encoded in either UTF-8 or WTF-8 https://simonsapin.github.io/wtf-8/ Future standards or non-standard systems MAY use different encodings; conformant implementations MUST NOT alter the sequence of bytes. They MAY perform a validation check and add additional headers.

The Rust standard library has contortions in order to support invalid UTF-16. To do this it uses an encoding called "WTF-8"[0], which is an extension to UTF-8. If the standard library dropped this requirement and only supported valid Unicode then it could simply use normal UTF-8 instead of WTF-8. [0]: https://simonsapin.github.io/wtf-8/

https://simonsapin.github.io/wtf-8/ (which I think you’ve found). People had occasionally used the label before for mojibake of various kinds, but the term was never popular under that meaning. Simon’s work is now a vastly more popular meaning.

WTF-16 makes the distinction that invalid sequences are allowed. * http://simonsapin.github.io/wtf-8/

WTF-8 and WTF-16 are a thing: https://simonsapin.github.io/wtf-8/ Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). Then WTF-8 is what you get if you naively transform invalid UTF-16 into UTF-8. It is a superset of UTF-8. This is very useful when dealing with applications like Java and Javascript that treat strings as sequences of 16-bit code points, even though not all such strings are valid UTF-16.

If there are surrogate pairs, I thought that was more of UTF-16 rather than UCS-2. Wikipedia says UCS-2 cannot go outside the BMP range (which emojis are). >UCS-2 differs from UTF-16 by being a constant length encoding[4] and only capable of encoding characters of BMP. https://en.wikipedia.org/wiki/UTF-16 There does seem to be some disagreement on this topic though. I wonder if it could be described as WTF-16. https://simonsapin.github.io/wtf-8/

UTF-16 is only a convention for file names on Windows. This is why we have things like WTF-8: https://simonsapin.github.io/wtf-8/

> The only reason I can see is in ensuring that text is losslessly convertible to other UTFs, particularly UTF-16 (which exists for historical reasons), but this just seems like a matter of when the information is lost (is it during conversion from string to UTF-16, or from bytes to string), not if it is lost. And you can even avoid losing that information at all with encodings like WTF-8: https://simonsapin.github.io/wtf-8/

> On the topic of unpaired surrogates, that a problem WTF-8 (https://simonsapin.github.io/wtf-8/) is intended to help solve. Yes. And it does so just fine. But you probably don't want your core string type to be that, so it's used as part of the "third way" where filenames are not strings, so that windows filename are relatively cheaply convertible to strings: by transcoding to wtf8 upfront, converting from filenames to strings is just UTF8 validation; and converting from UTF8 to filename is free. And likewise for "byte array" unix filenames.

On the topic of unpaired surrogates, that a problem WTF-8 (https://simonsapin.github.io/wtf-8/) is intended to help solve. The spec was created for Servo/Rust, but it's a sane general internal representation that should let people interact with platform APIs in a lossless manner.

https://doc.rust-lang.org/std/ffi/struct.OsString.html has it pretty clearly: on unixes it’s a bag of bytes, on Windows it’s the modified UTF-16 they’ve got going on. There’s a trick called WTF-8 that bridges some gaps, though that’s considered an implementation detail: https://simonsapin.github.io/wtf-8/ They’re conversions because they’re not UTF-8 in the first place, that is, they’re not String/str. The conversions are as cheap as we can make them. That language is meant to talk about converting from OsString to String, not from the OS to OsString.

I wonder if there is ever going to be an encoding that replaces UTF-8? Maybe WTF-8? https://simonsapin.github.io/wtf-8/