PyTorch Blog

I don't have any great recommendations and unfortunately my advice may be not what you want to hear. What I tell my students is "You don't need to know math to build good models, but you need to know math to know why your models are wrong." But this is even a contentious statement within the community. (Personally I'm more interested in exploring what we can build and understand rather than focusing on throwing more compute and data at problems. There's a lot of work to be done that does not require significant compute, but it isn't flashy and you'll get little fame. Every famous model you know has some unsung hero(s) who built the foundation before compute was thrown at the problem). I was previously a physicist and we similarly frequently express that you do not know the material unless you can do the math. Physicists are trained in generating analogies as they help communication but this sometimes leads to people convincing themselves that they understand things far more than they actually do. They say the devil is in the details, and boy are there a lot of details. (Of the science communicators, I'm happy those are the ones you mention though!) But do not take this as gatekeeping! These groups are often happy to help with the math and recommend readings. ML is kinda a while west and you can honestly pick a subdomain of math and probably find it useful, but I would start by making sure you have a foundation in multivariate calculus and linear algebra. As to paper…

This is close to home. Our approach to DevOps for ML Data is to use S3 as the git core and build immutable datasets and models on top of S3 object versioning. I wrote the piece below on "Versioning data and models for faster iteration in ML" earlier this year. The key idea is for every model iteration to be a pure function F(code, environment, data). Ideas welcome: https://medium.com/pytorch/how-to-iterate-faster-in-machine-learning-b…

> What other platforms are there for versioning data and recording data provenance? Quilt also versions data and data pipelines: https://medium.com/pytorch/how-to-iterate-faster-in-machine-learning-b… https://github.com/quiltdata/quilt (Python)

Have you checked out Cortex? It's an open source platform for deploying PyTorch models easily. I wrote an article for the PyTorch blog about it: https://medium.com/pytorch/how-to-build-production-software-with-pytor… GitHub: https://github.com/cortexlabs/cortex Full disclosure/shameless plug: I work on Cortex

Logged into my personal account for this one! I'm a lead author on a paper that explored exactly. It does enable faster training and smaller model sizes. For reference, you can get 80% accuracy on CIFAR-10 in ~30 minutes of CPU (not using crazy optimizations). There are open questions about scaling but at the time we did not have access to big compute (really still don't) and our goals were focused on addressing the original ViT's claims of data constraints and necessities of pretraining for smaller datasets (spoiler, augmentation + overlapping patches plays a huge role). Basically we wanted to make a network that allowed people to train transformers from scratch for their data projects because pretrained models aren't always the best solutions or practical. Paper: https://arxiv.org/abs/2104.05704 Blog: https://medium.com/pytorch/training-compact-transformers-from-scratch-… CPU compute: https://twitter.com/WaltonStevenj/status/1382045610283397120 Crazy optimizations (no affiliation): 94% on CIFAR-10 in <6.3 seconds on a single A100 : https://github.com/tysam-code/hlb-CIFAR10 I also want to give maybe some better information about ViTs in general. Lucas Beyer is a good source and has some lectures as well as Hila Chefer and Sayak Paul's tutorials. Also, just follow Ross Wightman, the man is a beast Lucas Beyer: https://twitter.com/giffmana/status/1570152923233144832 Chefer & Paul's All Things ViT:…