Henri Sivonen

OK, checked and Ruby does seem to use scalars. Well, unless you mess with encodings. Then it’s messy. So it’s probably better and worse than Python 3. You may not have seen this interesting article before: https://hsivonen.fi/string-length/. I agree with its assessment that scalars are really pretty useless as a measure, and Python and Ruby are foolish to have chased it at such expense. But seriously, I can’t think of any other popular languages that count by scalars or code points—it’s definitely not most languages, it’s a minority, all a very specific sort of language. “Most” encompasses well-formed UTF-8 (e.g. Rust), recommended UTF-8 but it doesn’t actually care (e.g. Go), potentially ill-formed UTF-16 (e.g. JavaScript, Java, .NET), and total-mess (e.g. C, C++).

It is up to the implementation. This is a good read on aspects of it: https://hsivonen.fi/string-length/

For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts

It is really an unfortunate over-engineering in the PNG format to include gamma correction. So many problems would have been avoided if PNGs just stored RGB values and let the renderer/OS/driver/screen decide how to display it. Trying to do gamma correction based on various guesses about screens just lead to jarring color inconsistencies. Here is great article about it. It is a few years old, so some of the problems are fixed now. https://hsivonen.fi/png-gamma/

The PyUnicode object is what represents a str. If the UTF-8 bytes are ever requested, then a bytes object is created on demand and cached as part of the PyUnicode, being freed with the PyUnicode itself is freed. Separately from that the codepoints making up the string are stored in a straight forward array allowing random access. The size of each codepoint can be 1, 2, or 4 bytes. When you create a PyUnicode you have to specify the maximum codepoint value which is rounded up to 127, 255, 65535, or 1,114,111. That determines if 1, 2, or 4 bytes is used. If the maxiumum codepoint value is 127 then that array representation can be used for the UTF-8 directly. So the answer to your question is that many strings are stored as UTF-8 because all the codepoints are <= 127. Separately from that, advancing through strings should not be done by codepoints anyway. A user perceived character (aka grapheme cluster) is made up of one or more codepoints. For example an e with an accent could be the e codepoint followed by a combining accent codepoint. The phoenix emoji is really the bird emoji, a zero width joiner, and then fire emoji. Some writing systems used by hundreds of millions of people are similar to having consonants, with combining marks to represent vowels. This - - is 5 codepoints. There is a good blog post diving into it and how various languages report its "length". https://hsivonen.fi/string-length/ Source: I've just finished implementing Unicode TR29 which…

I appreciate your reply, which I never expected in this situation. If my understanding is correct, your thesis is that Unicode should be hidden from application programmers as much as possible, much like the fact that GC hides memory management so to say. Not to say Unicode is bad or even shouldn't exist at all, but something like that it has to be abstracted away. This is a much more reasonable than most (quote-unquote) Unicode criticisms indeed. I'm not sure whether this is possible in the near future however, for reasons I'll work out here. ---- From perspectives of API consumers, most if not all programming languages have a suboptimal design for the human text. In fact the type name "string" itself is inappropriate, its name comes from the assumption that a human text is a string of symbols, which is not incorrect but not helpful either. A proper "human text" type (or a collection of them) should ideally be able to do the following: - An abillity to contain additional linguistic informations like locales, grammatical genders or numbers if possible. Some can be guessed, some can be retrieved from external contexts (e.g. HTTP `Accept-Language`), some have to be retrived with a consent. Any text operation should retain them if the corresponding text is also retained. - A language-aware formatter. For example `"Total: $ {n} files"` should automatically change "files" to "file" when `n` is 1. Moreover, `"Total: $ {n} $ {objectName}"` should do the same if `objectName` is…

> I’m not sure why I thought XHTML was completely dead in this way. Something’s niggling at the back of my mind, but I can’t place it. Browsers were always non-validating XML processors. Maybe you're thinking about the Well-formedness constraints? These constraints check if an XML document follows the XML metasyntax and any XML processor is required to fail [1] on errors in these contraints. And browsers do that. [1] https://www.w3.org/TR/2006/REC-xml11-20060816/#sec-conformance In practice most well-formedness violations were nesting errors like missing start or end tags, common when generating XML via text concatenation or templates, which in the dark times a lot of people do, even when they shouldn't . [2] That's why the HTML5 parsing algorithm is so complex and almost unimplementable. It's the ambition for error correction: It tries to generate a DOM out of any well-formed and unwell-formed documents. [2] https://hsivonen.fi/producing-xml/#nottext

> The argument for DRM is that it ships in a sandbox built by Mozilla. Actually, I was long since wondering how exactly that sandbox works, so if you have some more information about that, I'd appreciate it. Henri Sivonen's general explanation of EME and CDMs[1] tells the following: > A CDM could be bundled with the browser, downloaded separately, bundled with the operating system, embedded in hardware as firmware running in a second domain of computing (such as ARM TrustZone) or wired into hardware. EME leaves this aspect implementation-dependent. [...] EME does not specify the output abstraction for CDMs. It leaves open several options. The CDM could: - Merely perform decryption and hand back the encoded media (e.g. H.264) to the browser. [...] - Perform decryption and decoding and then work together with the GPU so that not even the operating system gets the opportunity to read the pixels back from the GPU. Meanwhile, Mozilla's implementation of EME seems to be substantially more restrictive[2]: > Firefox does not load [the CDM] directly. Instead, we wrap it into an open-source sandbox. In our implementation, the CDM will have no access to the user’s hard drive or the network. Instead, the sandbox will provide the CDM only with communication mechanism with Firefox for receiving encrypted data and for displaying the results. [...] in Firefox the sandbox prohibits the CDM from fingerprinting the user’s device. Instead, the CDM asks the sandbox to supply a…

That being said, I really don't see how processing UTF-8 is significantly more complex than processing, say, UTF-16. In both cases you need to handle continuation units for the extraction of Unicode code points. UTF-8 has 4 valid cases, one for each length, and many more invalid cases for each length (2-byte sequence missing trail byte, 3-byte sequence missing 1 trail byte, 3-byte sequence missing 2 trail bytes, 4-byte sequence missing 3 trail bytes, 4-byte sequence missing 2 trail bytes, ..., overlongs, UTF-8'd surrogates, overflow, etc.) Differences between implementations' treatment of error cases have lead to some security concerns; see https://hsivonen.fi/broken-utf-8/ and discussion at https://news.ycombinator.com/item?id=14451822 for an example. UTF-16 has two valid cases (one or two code units) and two error cases (lead surrogate not followed by trail surrogate, lone trail surrogate). It's more like a DBCS, except each code unit is 2 instead of 1 byte.

A tricky thing w/r/t browsers is that they are often really crappy at figuring out what color space a PNG image is supposed to be in. I think they often intentionally strip out the color space info from PNG images for the sake of helping PNG images blend smoothly into CSS / Flash / etc., for which they also don’t do correct color management. [The CSS spec says that colors should be interpreted as sRGB, but browsers don’t actually follow the spec.] As a result of this problem, it’s difficult to judge whether image resizing specifically is being done incorrectly, or whether the image is just being interpreted incorrectly at any size. You’ll notice if you open your first test image (http://www.4p8.com/eric.brasseur/gamma-1.0-or-2.2.png) in Apple’s Preview app, you get the correct behavior, but this is different from the behavior in Safari, for example, which shows something incorrect at every size. Stranger still, the behavior in Safari is roughly “correct” while resizing a browser window, but as soon as you let go of the mouse, it reverts to being incorrect. There are multiple possible ways to embed color space info into PNG images, and the PNG spec is actually really terrible and confusing on this subject which results in lots of inconsistencies from one implementation to another. If I remember correctly some of them are treated better by browsers than others, but I’m not sure if there’s a way to encode images such that it will be handled properly everywhere (I mean,…

> In pretty much every "Unicode string" library/language type I've seen, each individual codepoint is a "character" languages are actually really inconsistent on what they count as a unicode character: https://hsivonen.fi/string-length/ (I don't broadly disagree with you on unicode support, just linking an article relevant to that claim)

It seems the decision was made in the name of security: https://hsivonen.fi/no-encoding-menu/ > Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden). With the current approach, the parser needs to know of one flag to force chardetng, which the parser has to be able to run in other situations anyway, to run. > Elaborate UI surface for a niche feature risks the whole feature getting removed > Telemetry [...] suggested that users aren’t that good at choosing correctly manually. In other words, it's trying to protect users from themselves by dumbing down the browser. (Never mind that people who know what they are doing have probably also turned off telemetry...)

This was done with the HTML (5) parser. https://hsivonen.fi/html5-gecko-build/ There is the standards based code path and the quirks path.

It seems he did extensive background research to investigate and find optimal solution for this issue: https://hsivonen.fi/chardetng/ Goal was apparently to bring FF to feature parity with Chrome. Sad to see it's not working as well as intended. (Even sadder to see his person being attacked by some commenters below.)

Per https://hsivonen.fi/encoding-telemetry/, it looks like they run a special encoding detector on .jp sites, which is presumably designed specifically to choose between the various Japanese encodings (Shift-JIS, EUC-JP, etc). I think your example of emergency services is a bit hyperbolic. This is a feature that is really not often used, whose omission is not fatal, which often requires several tries to get right, and which is increasingly less useful thanks to gradual changes on the Web. Much more widely-used features like FTP and Flash have been deprecated; people howl and yell every time, but yet things still seem to work.

Seems like clickbait title. I have seen this issue few times over the years and it was always a game of "Which encoding should I choose?" Having an automated way to solve this seems pretty good to me. Especially if the solution works pretty much 100% of time (especially for non-ASCII languages, like Japanese or Chinese) - see paragraph Document-length Accuracy in https://hsivonen.fi/chardetng/

You can use FF as a sentinel byte internally (I think utf8proc actually does that?); given that FE never occurs, either, if you see the byte sequence corresponding to U+FEFF BYTE ORDER MARK in one of the other UTFs you can pretty much immediately tell it can’t possibly be UTF-8. (In general UTF-8, because of all the self-synchronization redundancy, has a very distinctive pattern that allows it to be detected with almost perfect reliability, and that is a frequent point of UTF-8 advocacy, which lends some irony to the fact that UTF-8 is the one encoding that Web browsers support but refuse to detect[1].) I don’t think there is any other advantage to excluding FF specifically, it’s not like we’re using punched paper tape. [1] https://hsivonen.fi/utf-8-detection/

Semi-related (about length of emoji "characters", not reversing): https://hsivonen.fi/string-length/ Previously discussed: https://news.ycombinator.com/item?id=20914184 https://news.ycombinator.com/item?id=26591373 As for this article & Python - as usual it is biasing towards convenience and implicit behavior rather than properly handling all edge cases. Compare with Rust where you can't "reverse" a string - that is not a defined operation. But you can either break it into a sequence of characters or graphemes and then reverse that, with expected results: https://play.rust-lang.org/?version=stable&mode=debug&edition=… (Sadly the grapheme segmentation is not part of standard library, at least yet)

https://news.ycombinator.com/item?id=23290506 https://coaxion.net/blog/2017/09/exporting-a-gobject-c-api-from-rust-c… https://hsivonen.fi/modern-cpp-in-rust/ http://hansihe.com/2017/02/05/rustler-safe-erlang-elixir-nifs-in-rust.…

I prefer this article: https://hsivonen.fi/string-length/ HN discussion: https://news.ycombinator.com/item?id=20914184)

"C++ has a very complex ABI, and the Rust ABI is not frozen. However, both C++ and Rust support functions that use the C ABI. Therefore, interoperability between C++ and Rust involves writing things in such a way that C++ sees Rust code as C code and Rust sees C++ code as C code." https://hsivonen.fi/modern-cpp-in-rust/

For an argument against using JSON-LD, see https://hsivonen.fi/no-json-ns/

...PNG24 for high quality, complex logos PNGs used to be a really bad choice for logos. PNG has a gamma correction value embedded in the image that's pretty much always wrong. It can be omitted, but then a browser can just make something up. If you want your image colors to match your CSS colors in older browsers, don't use a PNG. https://hsivonen.fi/png-gamma/

The fun thing is that XMLHttpRequest is much better designed as an API. The poor design of the script element objects caused problems when HTML5 tried to standardize it: https://hsivonen.fi/script-execution/

Reminds me of @hsivonen's wonderful "HOWTO Spot a Wannabe Web Standards Advocate": https://hsivonen.fi/wannabe/ Edit: Curious why this is being downvoted. Is humor (intelligent humor in this case IMO) really that frowned upon at HN? Or did someone think I was calling the parent commenter a "Wannabe Web Standards Advocate"? (I certainly wasn't. Quite the opposite.)

>'You can use "html", "html5", "html6.2" or "foobarbaz" as your doctype, the effect will be the same—they will all trigger standards compliant mode.' // You're wrong if your asserting that any doctype produces the same results globally but it probably doesn't matter unless you're targeting legacy UA. See e.g. "Appendix: Handling of Some Doctypes in text/html" at https://hsivonen.fi/doctype/ or Eric Meyer's http://archive.oreilly.com/pub/a/network/2000/04/14/doctype/index.html (for MacIE5 ... like I said it probably won't matter!).

The polyglot syntax gets weird in CDATA elements and you have to add a bunch of talismans to the code. If you don't want to accidentally break it you shouldn't be writing XML by hand or gluing it from strings (https://hsivonen.fi/producing-xml/), so you need to output only using polyglot-compatible XML+HTML serializer. That's a lot of work for case when maybe somebody will parse your markup as XML? All bots support HTML.

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." Like so many other things in Unicode, the correct answer is use-case dependent. (And for this reason, String iteration should be based on codepoints--it's the fundamental level on which Unicode works, and whatever algorithm you want to use to derive the correct answer for your purpose will be based on codepoint iteration. hsivonen's article (https://hsivonen.fi/string-length/), linked in this one, does try to explain why extended grapheme clusters is the wrong primitive to use in a language.)

> Could you please describe a bit more about the browser-CDM interaction and what is implemented there? I don't really know what this interaction looks like in non-Firefox browsers. Last I checked, the CDMs Chrome ships didn't work with Chromium, but I don't know whether that's still true, and I don't know whether the browser-side bits involved are implemented at all in Chromium or just in Chrome. Likewise, I don't know whether the CDM interaction bits in Safari are in the public WebKit repo or not. IE's source is not available, of course. In the case of Firefox, https://hacks.mozilla.org/2014/05/reconciling-mozillas-mission-and-w3c… really does cover most of the details. We put together an API that made sense on our (Firefox) end internally. We then worked with some CDM vendors to integrate their products, by building shims to convert the API their CDMs exposed to the API we wanted to be using internally. That's probably all I can say on the subject. https://hsivonen.fi/eme/ has a reasonably in-depth discussion of the way these bits fit together from someone who was much more intimately involved in this than I was. > Why do you think the browser-CDM interaction was left unspecified? Because the people writing the spec pushed back pretty explicitly on doing so, claiming that this would take too much time and overconstrain things too much in terms of both CDM and browser implementations. > Wouldn't a standard be beneficial to all parties, even CDM developers…