Hendrik Erz

Based on comments here, Azure Trusted Signing with a personal certificate and their GitHub action worked. This blog post below is great. It works through the process, including the truly weird/awful UX choices Azure makes. I archived it on wayback in case it ever disappears. https://www.hendrik-erz.de/post/code-signing-with-azure-trusted-signin…

I understand the sentiment, but this is why we have the free software sustainability crisis. https://www.hendrik-erz.de/post/open-source-has-a-sustainability-crisi… https://fsfe.org/freesoftware/sustainability/sustainability.en.html The time taken to type out Lorem Ipsum would be worth more than $5. I know you have other options in this case, but why would you prefer to waste your time than spend money?

Not sure why this got posted when the follow-up claws back much of it and leaves a pretty weak conclusion: https://www.hendrik-erz.de/post/should-you-use-sqlite

Perhaps this author shouldn't publish their advice about databases. I fully accept that people don't know everything, and I fully support them exploring new things and falling on their faces (if you don't, you aren't really trying). But I object to them offering advice at the same time: the following is from the followup article: https://www.hendrik-erz.de/post/should-you-use-sqlite The first argument [in the prior article] that is completely bogus is about the speed: I argued that, if you’re not careful, the access times of the database will be actually slower than to simply use the file system. Now, obviously there is a large flaw in this argument if you know anything about databases. Specifically, I never thought about the option to just create additional indices for columns I frequently addressed. !

It is confusing that the new link provided in the header was not posted instead: (Link: https://www.hendrik-erz.de/post/should-you-use-sqlite) In which the author clearly says: > About a year ago I fervently advocated against using SQLite for storing research data, but was since proven absolutely wrong. In this article I explain where and why I was wrong, and share the real reasons why I think we shouldn't use SQLite for research: A lack of skills and time. From the top of this article: > Update 29 Aug 2022: There is now an update to this article where I acknowledge problems with the arguments I brought forth below. Please read the update here: Should you use SQLite?

I didn’t read the whole thing but the intro seems even handed: https://hendrik-erz.de/post/lying-to-your-research-subjects-and-other-…

Article is from 2021, an update to the article published 2022: In this article I explain where and why I was wrong, and share the real reasons why I think we shouldn't use SQLite for research: A lack of skills and time. https://www.hendrik-erz.de/post/should-you-use-sqlite

What, nothing about Electron's unacceptable code bloat factor? Most disappointing! https://www.hendrik-erz.de/post/electron-bloated-web-and-trade-offs A GPT-4-on-GPT-4 flame war might be an incredible spectacle...

Suppose you wanted to train an LLM to do addition. An LLM has limited parameters. If an LLM had infinite parameters it could just memorize the results of every single addition question in existence and could not claim to have understood anything. Because it has finite parameters, if an LLM wants to get a lower loss on all addition questions, it needs to come up with a general algorithm to perform addition. Indeed, Neel Nanda trained a transformer to do addition mod 113 on relatively few examples, and it eventually learned some cursed Fourier transform mumbo jumbo to get 0 loss https://twitter.com/robertskmiles/status/1663534255249453056. And the fact it has developed this "understanding" as an ability to learn a general pattern in the training data enables it to compress. I claim that the number of bits required to encode the general algorithm is fewer than the number of bits required to memorize every single example. If it weren't then the transformer would simply memorize every single example. But if it doesn't have space then it is forced to try to compress by developing a general model. And the ability to compress enables you to construct a language model. Essentially, the more things compress, the higher the likelihood you assign them. Given a sequence of tokens say "the cat sat on the", we should expect "the cat sat on the mat" to compress into fewer bits than "the cat sat on the door". This is because the latter is far more common and intuitively more common…

I think NYT is going to win. LLMs are arguably compressed data archives with weird algorithms. The fact that they will regularly regurgitate verbatim quotes of training data is evidence of this, as are the guardrails that try to prevent this. The second piece of evidence is this paper explained here https://www.hendrik-erz.de/post/why-gzip-just-beat-a-large-language-mo… where instead of an LLM researchers used gzip compressed data as a model and it even beat trained LLMs. AI is a bit of a black box, but that doesn’t protect the operators of black boxes from rights violation suits. You can’t make a database of scraped copyrighted data and patented that querying that data is fair use. There needs to be law made here and the law just isn’t going to be “everybody can copy everything for free as long as it’s for model training”. Licensing will have to be worked out, actual laws and not just case law needs to be written. I have a lot of sympathy for lots of leeway for the open source researchers and hackers doing things… but not so much for Microsoft and Microsoft sponsored openai.