Meta AI Blog

AFAIK it's still an active area of research, and evidence from Meta AI [0] suggests that size and quality of data can let smaller (not necessarily less sophisticated) models do amazing things. But a lot of the advancements we're seeing right now are the result of more sophisticated models [1], and one person is doing some interesting work [2] around achieving transformer-level performance with other architectures. So it's not completely settled if more data is the answer. But it has a significant impact. [0] https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ [1] https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) [2] https://github.com/BlinkDL/RWKV-LM

> Doesn't feel like there's much competition for ChatGPT at this point otherwise, which can't be good. Facebook open sourced their LLM, called OPT [1]. There's not much else, and OPT isn't exactly easy to run (requires like 8 GPUs). I'm not an expect, so I don't know why some models, like the graphics generation we've seen, are able to fit on phones, while LLM require $500k worth of GPUs to run. Hopefully this is the first step to changing that. [1] https://ai.facebook.com/blog/democratizing-access-to-large-scale-langu…

I was curious and clicked on the [Dataset] link and got this: https://github.com/transpchan/Live3D/issues/1 and here's the paper review notes: https://github.com/transpchan/Live3D/blob/main/eccv.pdf which complains that it's too similar to https://anr-avatars.github.io/ and https://ai.facebook.com/research/publications/continuous-surface-embed… To me, this looks like great craftsmanship: They have built a working implementation where others failed. And they found a way to cheaply generate lots of training data. But sadly, research optimizes for novelty, not for usability.

Blog post: https://ai.facebook.com/blog/greater-creative-control-for-ai-image-gen… Paper: https://arxiv.org/abs/2203.13131 Meta AI's tweet: https://twitter.com/MetaAI/status/1547598454922153985?s=20&t=JxY8d…

Nope. The best minds work on whatever they're interested in (this includes making better ads): https://ai.facebook.com/research https://research.google/research-areas/

Facebook, PyTorch Mobile Team | US-Remote only | Full Time | Software Engineer (1 position available) We’re the PyTorch Mobile Team (https://github.com/pytorch/pytorch/) working on making PyTorch broadly available for a plethora of mobile devices both within Facebook and outside. We’re interested in motivated engineers in this space who are willing to work remotely (within the US). PyTorch is the most popular AI framework within the research community, and we’re working on making it production ready (especially on mobile devices). You can see here (https://ai.facebook.com/tools/pytorch/) that the journey for Mobile (marked experimental) has just gotten started, and you’ll be jumping on to the bandwagon as it starts to leave the station! Please drop us an email at (agaurav at fb dot com and dhruvbird at fb dot com), and we’d love to chat if this sounds interesting to you!

REALM is amazing, and Google's PEGASUS[1] taught itself to count between 2 and 5 (which is mindblowing). Basically the NLP groups from Google and FB[2][3] are always worth watching. [1] https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html [2] https://ai.facebook.com/blog/covost-v2-expanding-the-largest-most-dive… [3] https://ai.facebook.com/blog/introducing-a-new-large-scale-dynamic-dat…

Awesome! Computational photography is fascinating. A while back, I wrote some OpenGL shaders to do the same thing, and got stuck when I wanted variable bokeh based on scene depth. This is easy when you have the depth map (say, in a renderer). I recall reading about how Apple achieved this using stereo disparity mapping, which is great if you have the hardware. I hadn't considered a neural net approach like yours for auto-generating a depth map, although I now that I look into it, Facebook recently published research on just that! https://ai.facebook.com/blog/-powered-by-ai-turning-any-2d-photo-into-…

The search algorithm shares a lot in common with our Pluribus poker AI (https://ai.facebook.com/blog/pluribus-first-ai-to-beat-pros-in-6-playe… but we added "retrospective belief updates" which makes it way more scalable. We also didn't use counterfactual regret minimization (CFR) because in cooperative games you want to be as predictable as possible, whereas CFR helps make you unpredictable in a balanced way (useful in poker). The most surprising takeaway is just how effective search was. People were viewing Hanabi as a reinforcement learning challenge, but we showed that adding even a simple search algorithm can lead to larger gains than any existing deep RL algorithm could achieve. Of course, search and RL are completely compatible, so you can combine them to get the best of both worlds, but I think a lot of researchers underestimated the value of search.

It's only going to get better. There's just so much code out there to learn from. PL grammar is a lot more machine-readable than natural language. https://ai.facebook.com/blog/aroma-ml-for-code-recommendation/ https://visualstudio.microsoft.com/services/intellicode/

Despite the luck/noise in Poker, there are reasonable measures of performance, and while I'm not an expert in this area, the bot seems to be doing very well (see paper for details). Poker is not a "crap game for AI" it's actually quite a good game. It's a very simple example of a game with a lot of randomness (a feature not a bug) and hidden information that still admits a wide variety of skill levels (expert play is much better than intermediate play is much better than novice play). This is a great accomplishment. More links for reference: https://ai.facebook.com/blog/pluribus-first-ai-to-beat-pros-in-6-playe… https://science.sciencemag.org/content/early/2019/07/10/science.aay240…

I remember reading when Facebook announced Aroma to auto suggest common coding patterns to FB engineers[0]. I haven't used much for code completion before but this idea sounds really interesting. It sounds like the IntelliCode implementation sources suggestions from repos with a higher number of stars so hopefully that helps with noise. For JS I'm sure the variability in style is pretty big. For people that have been trying IntelliCode for a while does it help much with development? Does it have the ability to suggest larger code snippets or only word by word? [0]https://ai.facebook.com/blog/aroma-ml-for-code-recommendation

Facebook demonstrated a version of this recently https://ai.facebook.com/blog/aroma-ml-for-code-recommendation/.

> ChatGPT competitors of EU origin? ChatGPT is not research. If you want real research: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ Edit: end paper is https://arxiv.org/abs/2302.13971 > Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

The research paper "Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?" that was just released presents a comprehensive study on visual representations for embodied AI. Authors curated CortexBench, which includes 17 tasks spanning locomotion, navigation, dexterous, and mobile manipulation, across a range of environments and agents, and learning conditions. Existing pre-trained visual representations (PVRs) were evaluated on CortexBench, but no PVR was universally dominant, and an artificial visual cortex does not already exist. Over 4,000 hours of egocentric videos from 7 different sources and ImageNet were combined to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Scaling dataset size and diversity does not improve performance universally, but does so on average, contrary to findings in prior work. The team's strongest model, VC-1 (adapted), was competitive with or outperformed the best prior results on all benchmarks in CortexBench. This is the largest and most comprehensive empirical study to date of visual representations for embodied AI, which required over 10,000 GPU-hours of training and evaluation. VC-1 is open-sourced: https://github.com/facebookresearch/eai-vc/. Relevant links: Website: https://eai-vc.github.io/ Paper: https://ai.facebook.com/research/publications/where-are-we-in-the-sear…

Thank you for the thoughtful questions! 1) We believe that timing is a critical piece of this opportunity. With the recent media buzz around ChatGPT, we have found that leadership in companies large and small are actively considering how to best make use of LLMs in their business. The problems we've identified emerged as clear patterns across hundreds of calls with companies that are either currently managing LLM-powered features in production, or aspiring to. The level of interest was much smaller just 6 months ago, has grown quickly, and we anticipate it to grow only more in the near future. 2) We agree that with OpenAI's current dominance in the space, being provider-agnostic is not top of mind for most at the moment. We are betting that this will become increasingly important as the space evolves. We are already seeing Google investing hundreds of millions in Anthropic (https://www.bloomberg.com/news/articles/2023-02-03/google-invests-almo… Google working on their own LLMs (e.g. BARD), and Facebook launching their own LLM (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/). We expect this to become an increasingly competitive space and hope to provide companies with the tools needed to effectively evaluate their options.

Firstly, it is now pretty much exactly two years since OpenAI announced GPT-3 and it is great to see that access to these models – for scientists in particular – is becoming increasingly commonplace. I should also note that Hugging Face really have taken a “big tent” stance on the development of this model and their business in general. This is refreshing given that no other entity capable of raising the funds and manpower to develop these kinds of models have taken a similar stance (despite much deeper pockets…). Yes, Hugging Face is running on venture capital, but at least for now I see no reason to be cynical about their efforts that in every way appear genuine to me. Still, it is not all great. OpenAI, DeepMind, Google, etc. continues to produce models that are completely closed and thus can never be the subject to analysis from the rest of the community. Models which all dwarf GPT-3 at this point. My fear is that those of us arguing for access will continue to play catch up – maybe even forever. However, while BLOOM is more open than say FAIR’s OPT [1] – which had the gall to call itself both “open” and “democratising” despite restricting commercial usage in its “open” license – I believe there is a discussion to be had about how they (and many others in the community) use the word “open”. [1]: https://ai.facebook.com/blog/democratizing-access-to-large-scale-langu… While I am not a lawyer, I am somewhat familiar with licenses and have gone…

> .. I still believe that ranking/recommendation models should be open. Today, we don't know how social media recommend posts to us, we don't know if it is biased, or good for some sub communities. There are [1] some engineering blogs where we can see that is based on some research ideas (e.g. embeddings) are prune to bias [3] and "rich gets richer" phenomena. There should be an open marketplace where institutes/researchers submit open & explainable ranking/recommender systems. This is how you democratize access to such social platforms, but the business model opposes such idea to make it reality. There is a large body of research in the area of explainable recommendation systems/Explainable AI. There are regulations today to use Explainable AI systems in healthcare field, but few of them in areas that impact our mental health (e.g. the use of Social Media). [1]: https://ai.facebook.com/blog/powered-by-ai-instagrams-explore-recommen… [2]: Explainable Recommendation: A Survey and New Perspective, https://arxiv.org/abs/1804.11192 [3]: https://aclanthology.org/P19-1162v2.pdf

Are these numbers the energy to train a model? The whole point of these new NLP models is transfer learning, meaning you train the big model once and fine-tune it for each use case with a lot less training data. 5 cars worth of carbon emissions is not a lot given that it is a fixed cost. Very few are retraining BERT from scratch. EDIT: The other two points are also disingenuous. * "[AI models] will also fail to capture the language and the norms of countries and peoples that have less access to the internet and thus a smaller linguistic footprint online. " NLP in "low resource" languages is a major area of research, especially because that's where the "next billion users" are for Big Tech. Facebook especially is financially motivated to solve machine translation to/from such languages. https://ai.facebook.com/blog/recent-advances-in-low-resource-machine-t… * "Not as much effort goes into working on AI models that might achieve understanding, or that achieve good results with smaller, more carefully curated datasets (and thus also use less energy)." This is also a major area of research. Achieving understanding falls under the purview of AGI, which itself carries ethical and safety concerns. There are certainly research groups working toward this. And reducing parameter sizes of big networks like GPT-3 is the next big race. See https://news.ycombinator.com/item?id=24704952

It's interesting watching these attempts to understand Transformers. Are they Graph networks? Are they Hopfields networks? Are they convolutions? There was this research: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the… Turns out, GPT-like architectures appear to use the same representation throughout all the layers. So you can use the final head layer as a lens to see what words the network is thinking of as it ... "thinks". That's a bit contrary to what I imagined a GPT-like architecture was doing. I would have assumed that its embedding of ideas changed throughout the network, only reaching a sensible embedding near the end. At least from the perspective of the head layer, that doesn't appear to be the case. Facebook's research demonstrates that the MLP layer can be dropped in favor of more attention over learned knowledge vectors: https://ai.facebook.com/blog/making-transformer-networks-simpler-and-m… > Reading new Transformer papers makes me feel that training these models requires something akin to black magic when determining the best learning rate schedule, warmup strategy and decay settings. I found GPT-*'s schedule straightforward. In fact, most training schedules these days are "boring". Adam with warmup and either a linear or cosine decay. CNNs have been doing that for awhile, and the hyperparameters now are fairly robust. If you get within an order of magnitude you'll land within a few percent of…

17 feb 2023, https://twitter.com/llama_index/status/1626387226639888385 'Today, we’re kicking off a rebrand of @gpt_index to : LlamaIndex ' Meta announcing LLaMa 24 feb 2023 https://ai.facebook.com/blog/large-language-model-llama-meta-ai/

Sorry, but this is just incorrect. Go players have gotten stronger over time overall [0][1], and AI discovered many new ideas that all top pros have incorporated into their game-play (idk how to give a source for this, it's just very well known in the Go community that the style of play changed drastically in response to AlphaGo, and absolutely everyone trains with AI these days). [0]: "The sudden overall increase in agreement in 2016 also reinforces the belief that the introduction of powerful AI opponents has boosted the skills of professional players." https://ai.facebook.com/blog/open-sourcing-new-elf-opengo-bot-and-go-r… [1]: https://www.goratings.org/en/

Original release from Meta (April 17): https://ai.facebook.com/blog/dino-v2-computer-vision-self-supervised-l… Demos: https://dinov2.metademolab.com/

The unicorn example from MSFT's Sparks of AGI paper, https://arxiv.org/abs/2303.12712, probably counts a novel problem. As another example, ImageBlind, https://ai.facebook.com/blog/imagebind-six-modalities-binding-ai/, learned modality translations explicitly not in its training set such as text -> sound.

Code: https://github.com/facebookresearch/fairseq/tree/main/examples/mms Blog Post: https://ai.facebook.com/blog/multilingual-model-speech-recognition/ Paper: https://research.facebook.com/publications/scaling-speech-technology-t…

Here's the blog post with video of it working: https://ai.facebook.com/blog/multilingual-model-speech-recognition

Just a little push back here, I think you strike on the right theme where a programming language could fill this gap. However, I wonder if new domain specific languages will eventually be the more elegant solution. Think Modular's Mojo [1] or Meta's KNYFE [2] mentioned earlier this week. [1] - https://www.modular.com/mojo [2] - https://ai.facebook.com/blog/meta-training-inference-accelerator-AI-MT…

Additionally, here's a link the blog post about it, which I feel might actually be a better place for this post to link to instead of Verge's coverage of it: https://ai.facebook.com/blog/imagebind-six-modalities-binding-ai/

https://ai.facebook.com/blog/roberta-an-optimized-method-for-pretraini… As just one example, which was (is?) one of the most used flavors of transformers from the BERT era.

Facebook has been building out one of the most powerful GPU clusters on the planet [1] for AI research, since at least 2021. This blog post in Jan of 2022 says they were already doing NLP research on it "with the aim of one day training models with trillions of parameters.". Their 2017 cluster, had 22,000 NVIDIA V100s. [1] https://ai.facebook.com/blog/ai-rsc/