James Betker

Nice to see this - also, looks like one of the primary authors is jbetker, who built the best open source TTS model that I've seen yet (TorToiSe): https://nonint.com/2023/09/23/dall-e-3/ https://github.com/neonbjb/tortoise-tts

It's going to get worse. Here's a great primer on the problem we're facing by a Google -> OpenAI engineer. "I think that being able to automatically produce arbitrary content that is indistinguishable from human-generated content at scale is the death knell of the web as we know it." https://nonint.com/2023/03/09/gpt-might-be-an-information-virus/

You might be interested in http://nonint.com/. He made a number of quite detailed blogposts on his 2 gpu machines (both 8x 3090's), including racks, power delivery etc

Check out https://nonint.com/static/tortoise_v2_examples.html it's also non-realtime, provides great quality and it's open source.

The best is probably tortoise but you have to run it yourself https://github.com/neonbjb/tortoise-tts here are some demos https://nonint.com/static/tortoise_v2_examples.html

Be sure to check out the impressive demo clips! https://nonint.com/static/tortoise_v2_examples.html I wish there was a better link than this reddit post, but it's actually the best overall explanation of the project. This is the most impressive voice mimicking text-to-speech engine I've heard yet. Done by a solo developer on a home training rig.

The following blogpost by an OpenAI employee can lead us to compare patterns and transistors. https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ The ultimate model, in his (author's) sense, would suss out all patterns and then patterns among those patterns and so on, so that it delivers on compute and compression efficiency. To achieve compute and compression efficiency, it means LLM models have to cluster all similar patterns together and deduplicate them. This also means successively levels of pattern recognition to be done i.e. patterns among patterns among patterns and so on , so as to do the deduplication across all hierarchy it is constructed. Full trees or hierarchies won't get deduplicated but relevant regions / portions of those trees will, which implies fusing together in ideas space. This means root levels will be the most abstract patterns. This representation also means appropriate cross-pollination among different fields of studies further increasing effectiveness. This reminds me of a point which my electronics professor made on why making transistors smaller has all the benefits and only few disadvantages. Think of these patterns as transistors. The more deduplicated and closely packed they are, the more beneficial they will be. Of course, this "packing together" is happening in mathematical space. Another thing which patterns among patterns among patterns reminds me of homotopies. This brilliant video by PBS Infinite Series is amazing. As…

What may be relevant in this context is the observation that models ultimately converge on their dataset: > trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point. (…) > This is a surprising observation! It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else. Everything else is a means to an end in efficiently delivery compute to approximating that dataset. > Then, when you refer to “Lambda”, “ChatGPT”, “Bard”, or “Claude” then, it’s not the model weights that you are referring to. It’s the dataset. https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ – Is scraping good enough? Can we do without curated datasets? (This would be the most significant encyclopedic effort in human history – and we are by no means ready for this.) Edit: It may be worth remembering the Mundaneum, https://en.wikipedia.org/wiki/Mundaneum

Fair push back, but I do think the LSTM vs Transformers point kinda supports my position in the limit, not refutes. Once the compute bottleneck is removed, LSTMs scale favourably. https://arxiv.org/pdf/2510.02228 (I believe there's similar work done on vanilla LSTMs, but I'd have to go digging) So the bottleneck was compute. Which is compatible with 'data or compute'. But to accept your point, at the time the algorothmic advances were useful/did unlock/remove the bottleneck. A wider point is that eventually (once compute and data are scaled enough) the algorithms are all learning the same representations: https://arxiv.org/pdf/2405.07987 And of course the canon: https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ http://www.incompleteideas.net/IncIdeas/BitterLesson.html Scaling compute & data > algorithmic cleverness

Don't end to end trained models already do this to some extent? Like raising the pitch towards a question mark, like a human would. TortoiseTTS has a few examples under prompt engineering on their demo site: https://nonint.com/static/tortoise_v2_examples.html

Most of these are just an EPYC server platform, some cursed risers and multiple PSUs (though cryptominer server PSU adapters are probably better). See https://nonint.com/2022/05/30/my-deep-learning-rig/ and https://www.mov-axbx.com/wopr/wopr_concept.html.

I dare say RAG with vector DBs is underwhelming because embeddings are not underrated but appropriately rated, and will not give you relevant info in every case. In fact, the way LLMs retrieve info internally [0] already works along the same principle and is a large factor in their unreliability. [0] https://nonint.com/2023/10/18/is-the-reversal-curse-a-generalization-p…

In the spirit of hackernews, one can also build a rig such as this one: https://nonint.com/2022/05/30/my-deep-learning-rig/ and land a job at OpenAI as a side effect. Of all places, I wasn't expecting such pushback here

ANNs aren't top down either though. The architecture isn't as important as people think. https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ Even then, the architectures we use are essentially stumbled on. This is alchemy, not modern chemistry.

i suppose its not news, but there was a long thread about it but it came from this post from june: https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/

https://nonint.com/2023/07/05/on-the-efficiency-of-human-intelligence/ https://nonint.com/2022/05/30/my-deep-learning-rig/

James Betker in the tortoise-tts repo, which is similar, says he spent $15k for his home rig. I'm not finding right now how long it took to train the tortoise model but feel like I read him say weeks/months somewhere. Obvs all kinds of variations depending on coding efficiency and dataset size, but another datapoint https://nonint.com/2022/05/30/my-deep-learning-rig/ https://github.com/neonbjb/tortoise-tts

The developer behind Tortoise is experimenting with using diffusion for music generation: https://nonint.com/2022/05/04/friends-dont-let-friends-train-small-dif…