DataGenetics

Just today I learned that Nick Berry of https://datagenetics.com/ passed away recently. I guess that is slightly off topic but I wanted to share, I’ve been following his excellent blog for many years.

I write the blog: http://DataGenetics.com 1. Process: I keep a task list (using google tools) with a list of topic ideas. When I think of something I want to write about, or find interesting links, I place them there. There's typically about two dozen ideas at any time incubating. 2. Start: I started the blog about a dozen years ago as a way to add depth to my website that was the front-end for my consulting business. I stopped consulting shortly after starting, and am now retired, but I keep the blog up on that site; I've built a following. It was easy to start. Ideas are easy; the hard part is finding the free time to write! 3. Motivation. I love doing it. I get pleasure from writing. When I was a kid, I loved reading the works of Martin Gardner. I hope my blog could be seen some way as an homage to his works. 4. No real struggles, but I made a conscious choice to not enable comments on my site. Comments will get you down. Everyone is a critic. Jerks will say "that's not special, I could have don that" (hmmm, but you didn't). My email address is not hard to find on the site if you want to get in contact with me. Don't fret too much about publishing; do what is right for you. There is a whole spectrum of 'advice' from people saying regular rapid cadence (even if you have nothing to say), through to only publish gems. I typically average about one post a week. Some of my posts are lightweight, others are heavyweights.

> I'd like to find a way to repair QR codes in a reliable way. Consider reading up on https://www.nayuki.io/page/creating-a-qr-code-step-by-step And I highly recommend QRazyBox for manual painting with automatic decoding: https://merri.cx/qrazybox/ , https://github.com/Merricx/qrazybox > that begins with collecting a bunch of broken and corrupted QR codes Some ideas to consider: http://datagenetics.com/blog/november12013/index.html

I love DataGenetics, lots of interesting puzzles like that on there. There's a very unique style to all the visualizations and the solutions/analyses are always clear. Sadly, there was a post by the author in June 2019 about being diagnosed with Stage IV cancer [0]. There have been no posts since July 2022. I sincerely hope that's just because Nick doesn't have the time to blog anymore. [0] https://datagenetics.com/blog/june12019/index.html

Side note: DataGenetics has been my favorite blog in mid-2010s. Lots of great posts: http://www.datagenetics.com/blog.html

With a link to a great data story that is easy to miss: https://www.datagenetics.com/blog/september32012/index.html

first place I found this was https://datagenetics.com/blog/august32013/index.html which is still a good read.

And in fact Data Genetics is s great blog with loads of interesting content on maths, statistics and simulation. One that I've used as inspiration for a programming class I was teaching is his analysis of Snakes and Ladders: https://datagenetics.com/blog/november12011/index.html

It's worth mentioning that many smartphone users upload content to Youtube seemingly mistakenly and unaware that they are doing so (also known as fat-fingering[0]). A quick search for: DSC001.mp4 And then filtering that by recently uploaded always yields interesting results. For those that don't know, `DSC-XXX` is a standard naming scheme for digital cameras. More on the default naming scheme in the following link[1] [0] https://www.urbandictionary.com/define.php?term=fat-fingering [1] https://datagenetics.com/blog/december22012/index.html

Monty Hall himself wrote to the Harvard C.S. Professor Lawrence Denenberg questioning the counter-intuitive logic: > https://stats.stackexchange.com/questions/373/the-monty-hall-problem-w… ____ The topic of the counter-intuitive nature of probability reminds of Newton's letter to Pepys - "In 1693, Isaac Newton answered a query from Samuel Pepys about a problem involving dice. Newton’s analysis is discussed and attention is drawn to an error he made." Here is the classic Newton-Pepys Problem http://www.datagenetics.com/blog/february12014/ Here is the Newton-Pepys problem explained by Professor Joe Blitzstein in the Harvard class Stats110: https://www.youtube.com/watch?v=P7NE4WF8j-Q&feature=youtu.be... Here is further discussion about the logical error Newton made in his solution: http://arxiv.org/pdf/math/0701089.pdf

1. How many people do there need to be in a room so the there is a greater than 50% chance of at least two of them sharing the same birthday? 2. How many numbers we have to draw from 365 so the there is a greater than 50% chance of at least two of them are the same? 3. How many numbers we have to draw from X so the there is a greater than Y% chance of at least Z of them are the same? I think X,Y,Z are enough parameters: Drawing from X=1000 numbers, what is the chance Y that Z=(5,6,7,8...) is the same? Sorry, I'm not a matematician, just some breakfast ideas ;) Edit: Inspired by http://datagenetics.com/blog/february72019/index.html

For some mental fun, try these values for the angle and see what observations you can make about the results: 76.34541 126.869896 137.50777 However, I find it a bit bothersome that the input field doesn't properly accept floats, and the floats get rounded unnecessarily. Please don't do that, or have an option to customize it. Also, you miss the more obscure trigonometry values: http://datagenetics.com/blog/march22017/index.html Namely, to quote from the above page: Versine, Vercosine, Coversine, Covercosine, Exsecant, Excosecant, Haversince, Havercosine, Hacoversine, Hacovercosine You might also wish to expand this by adding, in addition to the already shown degrees & radians, the following angle measures: * Gradian * Turns * Arcminutes/Arcseconds

You reminded me of a really great post from the Data Genetics blog about estimating the amount of unseen errors in a document based on the errors actually found by independent reviewers. [0] It's such a simple idea but seems incredibly powerful. [0] http://datagenetics.com/blog/december12015/index.html

Two things: 1) You don't take your number when you find it. I guess my brain assumed "find" meant "find and take with you to prove you found it." Perhaps a better term might be "encounter." 2) Here's a much better explanation (thanks micaeked): http://datagenetics.com/blog/december42014/index.html. Essentially, the algorithm works because following numbers to drawers will eventually create a loop (e.g., drawer 2 points to 4 points to 6 points to 2). By starting with the drawer with your number, you're guaranteed to be in a loop with your number. The only question is whether your loop is less than 50 drawers long... Furthermore, this confirms that there IS no transfer of information from previous prisoners in any way.

Data Genetics did a great job doing a layman's write up on Benford's law, which is still one of the most fun math things to tell my family. Here it is: http://www.datagenetics.com/blog/march52012/index.html

"AFAIK it's just an urban myth" Your doubt tells us that your PIN is not a palindrome. The second most common PIN is 1-1-1-1, followed by 0-0-0-0 (says http://www.datagenetics.com/blog/september32012/ ), and looking at the top 20 numbers, over 10% of the people have a palindromic PIN. (If selected randomly, that should be 2%.) So either the police get called a lot, or there's some special flag that say "palindrome PIN - false alarm" (and it must be supported for international cards) ... or it's simply not true.

This may actually be a very good idea for some cases. If the stuff being protected by the key is, for example, the type of data that someone would otherwise keep in a personal fire-safe, then simply printing out the keys and keeping them in that fire-safe wouldn't change the type of security being provided. It might be nice if we had some type of highly-reliable (like the redundancy in QR codes[1]), so the process of printing out a key and re-scanning it was easy and trustworthy. [1] http://datagenetics.com/blog/november12013/index.html

This reminds me of one of their earlier posts about Calculus and monsters http://datagenetics.com/blog/october12013/index.html It's amazing to be reminded about how much math you forget after High School and never need to use again :(

Is there a good reason to not have a streaming update protocol for DNS? Surely it wouldn't be that difficult to prefetch the top N million DNS queries, then subscribe to a stream to keep them updated as TTLs expire? This would give you both privacy and speed (for the majority of queries) IPv4 address = 4 bytes Avg domain length = ~14 characters (http://datagenetics.com/blog/march22012/) So ~20MB per million domain names (assuming 1 IPv4 address per domain name)

I thought it was great! One nitpick: the word "Acceleration" is mis-spelled in the first section (http://www.datagenetics.com/blog/march42014/eq0.png).

Read the following, and see if you haven't changed your mind! http://www.datagenetics.com/blog/january42012/

I have to say, I've really liked some of the posts on this blog. The comparison of monte carlo and markov models of Chutes and Ladders[1] is very intuitive. I think it's a great way of understanding forward vs backward statistical reasoning without falling into the usual borderline religious divides. [1]: http://www.datagenetics.com/blog/november12011/index.html

An excellent visual tour of Benford's Law: http://www.datagenetics.com/blog/march52012/index.html My understanding of this law is that any collection of real-world data/metrics should follow this distribution pattern. There may be exceptions in which there are other factors influencing the data, but most should follow this "law". In that case, I don't see what is surprising about volcanic eruptions following this law. I would have assumed it already did (well, I suppose you have to already know about this law first to assume that).

I'm a lawyer. A highly specialized lawyer. I will share what I know about this question. 1. This is a problem for me too. Even though I have been doing this for a long time there are a lot of situations where I am asked for a referral and I don't know anyone good. Sometimes I don't know anyone at all. 2. When this occurs the first default thought is "Go to a Juggernaut Law Factory". All of them claim to have the best people in the world in every possible specialty. You pay through the nose. You hold your nose and hope that the work product is adequate. Sometimes it is. Sometimes it is astonishingly good. Sometimes it's embarrassing as hell how badly the work is done. (The same goes for hiring the Big 4 accounting firms). 3. If you don't want to go that route, your next job is to identify your problem as precisely as possible so you can start to find your way down the referral chain until you find someone you want to hire. 4. I will make reference to my specialty to give you an example, but in any area of law the same concepts hold true. 5. I am an international tax lawyer. I do primarily inbound (foreign money, companies, and humans entering the United States) work. To find me it is not sufficient to specify "tax" as your problem. That's like saying "I need some code" in software land, I'm guessing. You need to be highly specific. "I have a U.S. company that will sell 50% of itself to a foreign investor. My foreign investor needs a U.S. tax advisor…

This is a really belated blogspam repost. Original: http://www.datagenetics.com/blog/september32012/

2580 (straight down the number pad) is a popular PIN because of that. source: http://www.datagenetics.com/blog/september32012/index.html#google_vign…

Actual article: http://www.datagenetics.com/blog/september32012/index.html Should be changed to this, rather than screenshot + link blogspam.

This seems to be the source material: https://www.datagenetics.com/blog/march42014/index.html Vox re-heating a gizmodo article[0] which re-publishes (with permission) the one above. [0]: https://gizmodo.com/why-roller-coaster-loops-are-never-circular-154906…

There's a decent blog article here with some more strategies for Battleship: https://datagenetics.com/blog/december32011/index.html

Nice post, learned new things about QR codes. I noticed in the wild when doing check-ins, unreliable QR codes have been due to glossy paper and/or their placement, either being too high or low causing distortion. https://datagenetics.com/blog/november12013/index.html illustrates the impact of different destruction cases I'm surprised I haven't seen NFC tags being used for covid check-ins as a secondary option to QR codes