Keras Blog

Isn’t this just a specific application of an autoencoder though? Because random noise isn’t learnable, it gets filtered out if you teach the network to compress/decompress a sample with itself as target. https://blog.keras.io/building-autoencoders-in-keras.html With that said, it’s amazing how effective this concept is for cleaning up scientific data. Many have also used variational autoencoders to take it one step further and also cluster data by latent space features. I’ve used it myself to uncover various groups of behaviors in time-series from other cellular processes.

I don't disagree, but good vs bad error messages can be the different between a project that people love and contribute to and a project that people find frustrating and bombard with issues. Keras is great about error message[1] and it's part of what makes Keras beloved (and heavily contributed to). [1] https://blog.keras.io/user-experience-design-for-apis.html

This reminds me Keras's creator wrote a blog on how to design APIs for the best user experience: https://blog.keras.io/user-experience-design-for-apis.html

My mind was blown when I found out how easy it was to get started using pre-trained Glove embedding in Keras. Took my Kaggle game up a few notches over-night. https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-mod…

Hey OP! I've been using it extensively and it's definitely more than good enough for most tasks you'll be doing. If you're going this route, I'd highly recommend following the Keras' authors guide on how to set up the instance and connect to it with a jupyter notebook: https://blog.keras.io/running-jupyter-notebooks-on-gpu-on-aws-a-starte… This will boost your productivity a ton in the long run. Best of luck!

Another great example is the last post on the keras blog [1] "Using pre-trained word embeddings in a Keras model". You take advantage of a large pre-trained network for a text classification task. The OpenFace face recognition library also offers this technique. You take advantage of their large pre-trained network for face embedding: transforming a face into features distinct enough for classification. You then train another few layers for recognizing your own samples. 1: https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-mod…

I also recommend reading @fchollet's guide on integrating Keras and TensorFlow, especially for those wanting to implement novel components at a lower level :) http://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tut…

interesting topic. Could you point me to some resources so I can learn more about it? The following is a comprehensive introduction and survey of recent work in the field: https://www.microsoft.com/en-us/research/wp-content/uploads/2017/10/pr… Regarding novelty, neural networks are excellent at interpolation but awful at extrapolation. What I mean by interploation is that the models learned by a neural network represent a dense region of cartesian space circumscribed by the neural net's training instances. The trained model can recognise and generate instances included in this dense region, including instances not included in its training set, by interpolation, but it is almost completely incapable of representing instances outside this dense region by extrapolation [1]. So basically what I'm saying above is that deep learning language model code generators can't extrapolate to programs that are not similar to the programs they've been trained on. Hence, no real novelty. But you can go a long way without such novelty hence my expectation that e.g. Copilot can be a nice boilerplate generator. ______________ [1] My standard reference for this is the following article by François Chollet, maintainer of Keras: https://blog.keras.io/the-limitations-of-deep-learning.html See also the second part of the series of articles where possible ways are proposed to address this limitation: https://blog.keras.io/the-future-of-deep-learning.html The latter is…

Keras still is the very best in terms of expressive models and end to end workflows. It leans heavily on the design idea that you should deliberately design for end to end use cases and all intermediate abstractions should be building blocks that serve precisely that purpose. This is discussed in [0] which IMO is something that deserves to be more widely talked about in software engineering. Lots of other disciplines of software engineering _say_ you should design this way, but in my experience it’s very rare no matter what discipline you’re in. Take TensorFlow itself. It’s a huge mess with no clear abstractions useful for end to end solutions. Just a hodge podge of disparate APIs and way too many underlying engineering concepts were elevated to abstractions for engineer (instead of user) convenience. Constraining design by end to end use cases is a remarkably robust and useful process. PyTorch is way better at having clean engineering abstractions than TensorFlow, but still falls short when things like “forward” or maintaining your own training loop and gradient metadata are necessary concepts for a practitioner’s end to end workflow. [0]: https://blog.keras.io/user-experience-design-for-apis.html

I don't have much opinion on the rest of your translation, but I think this gets it wrong: > Postmodernists make for shitty programmers To me one of the big differences with postmodernism is a deep skepticism for grand overarching theories and perfect understandings. I think that's an immensely useful trait for a programmer. The more I code, the more I treat any digital representation as a deeply distorted image of the reality I'm dealing with. The best I can do is have the distortions match my user's purpose. As Box wrote, "All models are wrong; some models are useful." The modernist continues to seek the perfect model. The postmodernist treats perfection as a fool's errand and ships something that works well enough. I have seen this problem over and over in my career; a lot of architecture astronautics strikes me as rooted in modernism. But perhaps a good example that everybody can look at is early concepts of AI vs today's machine learning. When I compare Cyc [1], which has spent 30 years trying to build the seeds of perfect knowledge versus today's deep learning, that strikes me as a postmodern shift. How does a convolutional neural network recognize a dog? We don't exactly know, not in a way that makes much sense to us. [2] But we don't care. The convnet has its own very bounded, limited version of "truth"; as long as we get a 90% match, we ship it and go out for beers. [1] https://en.wikipedia.org/wiki/Cyc [2]…

Definitely, whenever I come across informative / helpful error messages I'm able to move faster. There is a nice write up from the creator of Keras about this: https://blog.keras.io/user-experience-design-for-apis.html ...See "3 - Provide helpful feedback to your users."

In general the trend is toward more complex architectures, which increasingly blurs the line between modular nets and sparsely connected parts of a single model. IMO it's only meaningful to talk about modularity if the modules can be separately trained, or at least have their own loss functions. But of course its always useful to train or finetune the entire system end-to-end. A semi-recent success story is Tacotron 2, a speech synthesizer that has one model to convert text to a spectrogram, and a second model to convert that spectrogram into pcm (waveform) data. Also, Francois Chollet, the author of the popular deep learning library Keras, has some interesting thoughts on the future of ML along these lines that I find compelling: https://blog.keras.io/the-future-of-deep-learning.html

https://blog.keras.io/building-powerful-image-classification-models-us… This is a good start

This class of tasks is called sequence2sequence (sometimes seq2seq) in machine learning. It's a standard example used by many tutorials. Here's one for Keras: https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequenc…

Yes, I agree that the applications for word vectors are not made as clearly as it should be. One direct application is as the first layer of a neural network [1], which could be part of either a 1-dimensional convolution or a recurrent neural network. Using pre-trained word vectors is a form of transfer learning and allows for much more predictive models with smaller amounts of training data. [1] https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-mod…

I'm excited for self-driving cars and ML/AI tools that help professionals (e.g., drug discovery, treatment). The Keras lead developer wrote a post called "The future of deep learning"[1]. The podcast Partially Derivative has a good summary of it[2]. On the implementation side, I'm looking forward to distribution. Computers are all around us, sitting by idly – how can we put them to use? How can we make them secure? We're starting to see integration of ML models into mobile systems (with Apple's .mlmodel and ARKit). [1]:https://blog.keras.io/the-future-of-deep-learning.html [2]:http://partiallyderivative.com/podcast/2017/07/25/the_future_of_deep_l… [3]:http://papers.nips.cc/paper/4390-hogwild-a-lock-free-approach-to-paral…

The Keras blog also has a good post on using CNNs to label text from newsgroups with their respective newsgroups: https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-mod… It achieves 95% accuracy without many tricks, but the newsgroups themselves are quite different so it's not the hardest problem.

Noise removal? See this page (https://blog.keras.io/building-autoencoders-in-keras.html) and search for "Application to image denoising"

https://blog.keras.io/building-powerful-image-classification-models-us… is what you want.

I really liked those ones: - CS321 from Standford (outstanding material by Andrej Karpathy): http://cs231n.github.io/ - Hugo Larochelle videos: https://www.youtube.com/channel/UCiDouKcxRmAdc5OeZdiRwAg - Michael Nielsen tutorial: http://neuralnetworksanddeeplearning.com/ - Chris Olah blog: http://colah.github.io/ - Keras blog: https://blog.keras.io/ There is also this one (not really a tutorial though): - Goodfellow, Bengio, Courville: http://www.deeplearningbook.org/ Also, there were a lot of 'Ask HN' topics - take a look.

It would be nice to also have a way to visualize filters (eg. https://blog.keras.io/how-convolutional-neural-networks-see-the-world.…

Yes, you can use these models for fine-tuning (or feature extraction) on a new dataset. This tutorial would be a good place to start (esp. sections 2 and 3): https://blog.keras.io/building-powerful-image-classification-models-us…

The universal approximator approach is another big one that people lean on from the opposite direction (I suppose you could call it the "anti-deep learning" school of thought). Most people don't take the time to consider what a universal approximator actually is. Are humans universal approximators? Absolutely not. The mere proof that a nn with an infinite number of hidden nodes (which isn't practical, which could end the discussion there) can find a statistical mapping between any two input and outputs is hardly a benefit. There is something so much more to intelligence --- the ability to generalize well (aka not overfitting) is first and foremost. I'd push you to ask why you perceive that stacking RBMs produces "eerily human-like" results. I'd argue that they don't: http://blog.keras.io/how-convolutional-neural-networks-see-the-world.h…

For deep learning / neural networks, I think these are the best balance of theory and practice (not to mention great visualizations): http://cs231n.github.io More on the intuition/theory side for NNs, but Michael Nielsen’s book is highly recommended: http://neuralnetworksanddeeplearning.com For NLP, you can try this extremely slick interactive course on spaCy (highly recommended Swiss Army knife library for NLP) from package author Ines Montani: https://course.spacy.io The Keras tutorials are solid, here’s one on seq2seq models: https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequenc… For more cutting edge NLP, it looks like the fast.ai course covers the Transformer model (the basis of BERT, GPT-2, et al. This is a great overview of that architecture: http://nlp.seas.harvard.edu/2018/04/03/attention.html And PyTorch implementations of Transformer models can be found here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/REA… LMK if anyone wants more, I have a learning resource hoarding problem.