David Hagen

Interesting! Be sure to follow the link to the second post about what happened to the 2nd, 3rd, 22nd, and 23rd. It's simpler but still worth the read: https://drhagen.com/blog/the-missing-23rd-of-the-month/

This is a great start, but as you get more serious about NLP, keep in mind that a Web crawl from 2006 is not the same as ground truth about language. The top 100,000 words in this data include a lot of ephemeral spam. If you're looking for slang, it'll mostly be outdated. And words like "sitemap" and "Shockwave" are not as common throughout language as the Web alone would indicate. The later Google Books Ngrams [1] data sets are cleaner, but that comes at a cost. You lose spam, but also a lot of other uses of language, when you only consider text that has been published in print. It's all in the formal register. And then you also get correlated OCR errors, as in [2], and you still get data whose collection ended in 2008. So what am I suggesting you should use, if data from the Web is bad for some things and data from books is bad for others? Well, both of them, and a lot of other things too. An analysis that depends on word frequencies should use the consensus of many sources. I've been working on compiling together many of those sources, in many languages, in the Python package wordfreq [3]. Now I'm working on an update to include excellent corpus data just released by the OPUS project, including OpenSubtitles 2016 [4]. [1] http://storage.googleapis.com/books/ngrams/books/datasetsv2.html [2] http://drhagen.com/blog/the-missing-11th-of-the-month/ [3] https://github.com/LuminosoInsight/wordfreq [4] http://opus.lingfil.uu.se/OpenSubtitles2016.php

For 2, 3, 22, 23, read the second part: https://drhagen.com/blog/the-missing-23rd-of-the-month/. (short version: people used to sometimes write 2d, 3d, 22d, 23d.). This also explains why 12 and 13 don't show the deficit.

Very interesting. I've had a thought about a monthly "data preservation day" concept, (on the 11th, because: https://xkcd.com/1140/, and yes, I know: https://drhagen.com/blog/the-missing-11th-of-the-month/) A multiplatform archival tool would be useful.

Reminds me of how when xkcd looked at which days of the month were most common, the 1st, 10th, 11th, 21st and 31st were more or less common than they should have been due to OCR error: https://drhagen.com/blog/the-missing-11th-of-the-month/

Note that, the article being from 2015, the promised future post for the 23rd is already live at http://drhagen.com/blog/the-missing-23rd-of-the-month/ and differently fascinating.