Kaggle Blog

I don't have great links for this, but for something less technical, you might look at the blog posts from Kaggle competition winners. Here are a couple examples http://benanne.github.io/2015/03/17/plankton.html http://blog.kaggle.com/2016/04/13/diagnosing-heart-diseases-with-deep-… or check out what's available in a deep learning library like keras http://keras.io/preprocessing/image/ Sorry I don't have better academic references.

My personal preference is to learn by doing, and the best place I've found for this particular task is Kaggle (http://www.kaggle.com). They have a variety of datasets and scored data mining tasks, great forums for every level, code examples, and even a set of tutorials specifically for learning scikit (one of Python's machine learning libraries): http://blog.kaggle.com/2015/04/08/new-video-series-introduction-to-mac… I don't have any relationship with Kaggle other than being a semi-active user, but I really dig what they've got going. For a step-by-step approach, start with their blog posts and work on their "Getting Started" competitions. Everything you need is there.

On the 4th page towards the bottom, Kaggle says they will host the team's writeup "soon"; this was 26 days ago. I do not see it on the Kaggle blog [1]... still coming? Or did I miss it? (I scanned over the titles for the previous month and ran a search on BAYZ. Despite the topic I don't think the "Microsoft Malware Winners' Interview: 1st place, "NO to overfitting!"" post is related.) [1]: http://blog.kaggle.com/

>> ... however, the choice of algorithm mattered little... and the largest gains always came from introducing new features. This was also the collective opinion of top Kaggle competitors as described in a post[1] on NoFreeHunch. [1] http://blog.kaggle.com/2014/08/01/learning-from-the-best/

Subscribe to Data Science Weekly: http://www.datascienceweekly.org/ http://blog.zipfianacademy.com/ http://blog.cloudera.com/blog/category/data-science/ http://www.hilarymason.com/ http://mathbabe.org/ http://fivethirtyeight.blogs.nytimes.com/ http://blog.kaggle.com/ http://grepalex.com/ http://pulseblog.emc.com/category/big-data/ http://radar.oreilly.com/ http://flowingdata.com/ http://oreilly.com/data/newsletter.html http://www.gapminder.org/ http://mlcomp.org/

Kaggle has blog posts of many of the past winners: http://blog.kaggle.com/category/dojo/ For algorithms, just try whatever you know best/is fastest to implement. If you're using R I highly recommend the Caret package. For papers: the best place to get started is to begin browsing the forums or any similar contests, the community there is actually pretty awesome and will frequently post papers. After that google scholar (or even just google) for particular problems will yield nice results. Also checkout the wiki: http://www.kaggle.com/wiki/Home

The value there, IMO, is more the links than the discussion. It's not so much like HN, where the discussion itself is half (or more) of the value. But, for a quick, concise list of new links in those fields, I find those Reddits all very valuable. better choices are http://metaoptimize.com/qa http://blog.kaggle.com/category/how-i-did-it/ and following ml topics on quora Yes, those are excellent sites as well. There are also some good StackExchange sites that can help one get their fill of CS'y topics.

I recommend the video at http://blog.kaggle.com/2011/03/23/getting-in-shape-for-the-sport-of-da… I gave more recommendations at http://stackoverflow.com/questions/598726/overwhelmed-by-machine-learn… (Disclaimer: I work at Kaggle) EDIT: Fixed second link

An example of what I was aiming at is to use ML methods like deep nets to detect early-onset diabetic retinopathy. Diabetic retinopathy is the leading cause of blindness and at least 90% of new cases could be reduced with early detection and proper treatment. [1] Especially in third world countries where there is no eye doctor in sight, these cheap automated methods can be deployed on a mobile phone and achieve near-human expert level accuracy. [2] Then organizations like Watsi can use data science and predictive modeling to reduce fraud and get both detection and treatments to those most in need. [3] [1] https://en.wikipedia.org/wiki/Diabetic_retinopathy [2] http://blog.kaggle.com/2015/09/09/diabetic-retinopathy-winners-intervi… [3] https://dssg.uchicago.edu/ About IBM Watson, the entire thing is unfortunate, I completely agree. Their marketing department upsold IBM Watson for cancer treatment. But I know that a lot of great research scientists worked at IBM on Watson. What they were doing was legit advancing machine learning too. That's the thing about marketing: if IBM were to deploy 10.000 phones with a neural net to improve early detection of disease in a third world country, I probably won't even hear about it, and I work with ML. But for IBM Watson, everybody and their grandmother goes: That's the AI that beat Jeopardy. It's a thin line between being majorly successful in marketing and crossing the line into damaging your reputation and…

They are excellent questions and that indeed is exactly the goal of this course. I hope you try it out and let us know if you find the answers you need. For this particular question, a model that does localization and then integrated classification is called an "attentional model". It's an area of much active research. If your images aren't too big, or the thing you're looking for isn't too small in the image, you probably won't need to worry about it. And if you do need to worry about it, then it can be done very easily - lesson 7 shows two methods to do localization, and you can just do a 2nd pass on the cropped images manually. For a great step by step explanation, see the winners of the Kaggle Right Whale competition: http://blog.kaggle.com/2016/01/29/noaa-right-whale-recognition-winners… (There are more sophisticated integrated techniques, such as the one used by Google for viewing street view house numbers. But you should only consider that if you've tried the simple approaches and found they don't work.)

In a data mining / machine learning class I took in university a key point my teacher made was that of the 'No free lunch' theorem. In that class the theorem was offered on its face and I haven't yet read the original paper (just summaries and recounts) (http://web.archive.org/web/20140111060917/http://engr.case.edu/ray_sou… but the point of the theorem feels very related to the point the author makes. The thesis boils down to: 'not having seen the data, there is no universally superior algorithm'. The author's point of view seems to comport with this theorem. I haven't gone through the paper myself, so I can't argue for or against it. However, in my experience as a professional statistician that has used Random Forests, SVMs, Neural Networks, etc --- finding the right algorithm takes 'feature engineering' (http://blog.kaggle.com/2014/08/01/learning-from-the-best/) and close inspection of the data and basis function. Are the data fit well by rectangles? Then you'll want to use decision trees / random forests. Curvy? Give SVM a try. Do the data take intricate patterns? Look into boosting (fitting on residuals recursively). Feature engineering boils down to the old statistical phrase, 'live with the data.' That is, you actually need use your own mind and beliefs to organize inputs prior to feeding them into an algorithm. So in my experience, what seems to be true in Kaggle competitions, and the perspective of this…

Why Machine Learning is Kaggle competitions. I used an out-of-the-box algorithm, messed around a bit, and definitely did not make the leaderboard. Because that is not Kaggle competitions. Nearly everyone on the leaderboard is proficient in data analysis and machine learning. They all tried that out-of-the-box algorithm for their attempt. But they did not give up so easily. Understand the business problem If you want to predict flight arrival times, what are you really trying to do? This is not different from Kaggle competitions, this is a tip for performing better in Kaggle competitions. See also the GE Flight Quest: https://www.gequest.com/c/flight Those winners used industry-standard machine learning, optimization techniques, but also creative insights and hunches, like tweaking the target labels: "A next step is to ask, “What should I actually be predicting?”. This is an important step that is often missed by many – they just throw the raw dependent variable into their favorite algorithm and hope for the best. But sometimes you want to create a derived dependent variable. I’ll use the GE Flight Quest as an example: you don't want to predict the actual time the airplane will land; you want to predict the length of the flight; and maybe the best way to do that is to use the ratio of how long the flight actually was to how long it was originally estimated to be and then multiply that times the original estimate." - Steve Donoho -…

Thanks, I’d never thought of doing this before - found 2 resources to help me explore this approach: http://zmjones.com/make/ http://blog.kaggle.com/2012/10/15/make-for-data-scientists/

I don't think the paper highlighted enough that this was a Kaggle competition entry. The write up on that (from Google, who organized it and provided the data) was really interesting: http://blog.kaggle.com/2018/02/07/a-brief-summary-of-the-kaggle-text-n…

I’m sure they can. But they can also choose not to distinguish a high quality user from a low quality one when promoting their site. For example, this post - http://blog.kaggle.com/2017/06/06/weve-passed-1-million-members/ - talks about a million users. But does not say how many are low quality, inactive ones. Or even describe how they filtered out low quality users from their analysis.

Official announcements: Google: https://cloudplatform.googleblog.com/2017/03/welcome-Kaggle-to-Google-… (HN: https://news.ycombinator.com/item?id=13822635) Kaggle: http://blog.kaggle.com/2017/03/08/kaggle-joins-google-cloud/ (HN: https://news.ycombinator.com/item?id=13822727)

You should take a look at Kaggle.com. There are lots of interesting problems, but what really was enlightening to me, was to look at the solved contests, and see what the winning solution implemented. http://blog.kaggle.com/category/winners-interviews/

See also this Kaggle competition where the task was to predict impending seizures: https://www.kaggle.com/c/seizure-prediction The top contestants achieved an accuracy of around 0.82 AUC, which blew the top of previous efforts. http://www.npr.org/blogs/health/2014/12/10/369654830/a-crowd-of-scient… This blogpost reviews the two Kaggle competitions and interviews the top contestants on their methods: http://blog.kaggle.com/2015/01/26/american-epilepsy-society-seizure-pr… Very rewarding projects to hack on.

This competition is the first "Playground" one just for fun http://blog.kaggle.com/2013/09/25/the-playground/

Here's another (same comp, MCMC models took top 2) http://blog.kaggle.com/2012/12/19/a-bayesian-approach-to-observing-dar…

To see why Yahoo is doomed, let's compare and contrast Yahoo's recent acquisition of Summly with Google's DNNResearch. Summly - Sloppy mobile app that licenses technology from SRI. DNNResearch - Groundbreaking fundamental CS Research into Deep Neural Networks that won Merck's Kaggle Competition. Enough said. Edit links - http://techcrunch.com/2013/03/12/google-scoops-up-neural-networks-star… http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st…

Weka: http://www.cs.waikato.ac.nz/ml/weka/ Lets you test dozens of different algorithms on a given dataset, to see what works and what not, in like one evening. Also success stories from Kaggle competition: http://blog.kaggle.com/category/dojo/

Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician. source: http://blog.kaggle.com/2012/10/04/engineering-practices-in-data-scienc… Software development is important, but it's not everything. People choose what works and use it. Though I agree that spreadsheets won't always work on Big Data, there's no point in doing more if you can achieve it with less.

The paper from the same group last year on the topic is here: http://rsif.royalsocietypublishing.org/content/8/59/842.full#sec-9 They are using random forest out-of-sample error as a metric but doing feature selection before this step (see table 6). As far as I can make out from a quick reading they are essentially making the error described here: http://www-stat.stanford.edu/%7Etibs/sta306b/cvwrong.pdf and elegantly described in this recent blog post: http://blog.kaggle.com/2012/07/06/the-dangers-of-overfitting-psychopat… On a sample size of only 42 people, overfitting seems very likely.

The hype for neural networks is deserved. Some major contributions to the field resulted in increases in accuracy for fields like NLP, computer vision, structured data, machine translation, style transfer, etc. XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning. XGBoost is popular for structured data competitions on Kaggle. Even there: The winner is often an ensemble of XGBoost and Keras. And some structured data competitions are won by neural nets alone: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st… and https://www.kaggle.com/c/higgs-boson/discussion/10425 (Neural nets won the Higgs Boson Detection Challenge, where XGBoost was introduced) I'd say Tensorflow/Keras can handle a wider variety of problems, with the same, or improved accuracy, than tree-based methods can. NN's do well on structured problems (the domain of tree-based methods), but also own computer vision and, increasingly, NLP. I agree on the pitfalls of applying neural nets vs. forests. It is true that tree-based methods are academically a bit out of vogue: The exciting stuff is happening in the neural network space. You have more chance of getting published with deep learning (this used to be the other way around).

I agree (except for the first word), however I read the question with emphasis on "usual", as in, "What makes DNNs special?" There's pure performance (ex., in a Kaggle competition [http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st… or on a standard data set [http://yann.lecun.com/exdb/mnist/], [http://blogs.microsoft.com/next/2015/12/10/microsoft-researchers-win-i… ), but that's what makes any ML method better than another. I think the deeper awesomeness is that DNNs so good at Feature Learning from raw data. On vision, NLP, and speech problems [nice overview by Andrew Ng: https://m.youtube.com/watch?v=W15K9PegQt0] DNNs have achieved superior performance to the combination of expertly-engineered features + some usual ML algorithm. Where a "usual ML" pipeline might look like (1) engineer features through manual effort by studying raw data and the problem domain, (2) apply ML to those features, a new DNN pipeline might look like (1) Apply DNN to raw data. First off, removing the feature engineering step could be a huge savings in human time spent. Second, there's the potential to get a better answer (!) when you're done. But more than that, the DNN pipeline holds the promise of more regular, systematic improvement. We (as engineers) don't have to wait for a bright idea about how to construct a feature from the data. Instead, we can focus on (1) collecting more and…

The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs [3]. There are also lots of examples beyond classification - structured prediction such as image segmentation [4], predicting depth, surface normals, and pixels from only RGB [5], text to text translation [6], image captioning [7], Turing machines [8], speech synthesis [9], question answering (Q/A) [10][11], handwriting generation [12], playing Go [13], playing Atari [14], speech recognition [15], drawing things [16] and many more. [1] http://blog.kaggle.com/2015/07/27/taxi-trajectory-winners-interview-1s… [2] http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st… [3] http://arxiv.org/abs/1403.6652 [4] http://arxiv.org/abs/1502.03240 [5] http://arxiv.org/abs/1411.4734 [6] http://arxiv.org/abs/1409.0473 [7] http://arxiv.org/abs/1502.03044 [8] http://arxiv.org/abs/1410.5401 [9] http://arxiv.org/abs/1506.02216 [10] http://arxiv.org/abs/1507.04808 [11] http://arxiv.org/abs/1503.08895 [12] http://arxiv.org/abs/1308.0850 [13] http://arxiv.org/abs/1412.3409 [14] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf [15] http://www.sciencedirect.com/science/article/pii/S0893608014002007 [16] https://www.youtube.com/watch?v=Zt-7MI9eKEo

In addition to what bravura said, Geoff Hinton's group at UT has recently introduced a new approach to training neural networks that they call "dropout". You can read about it in their paper (http://arxiv.org/pdf/1207.0580.pdf) or watch Hinton describe it in a recent talk he gave at Google (http://www.youtube.com/watch?v=DleXA5ADG78). Roughly speaking, dropout training provides a strong regularizing effect through a sort of model averaging that is conceptually related to the well-known bagging approach from which random forests derive their power and flexibility. Dropout training has already produced state-of-the art results on several time-worn standard benchmarks and helped Hinton's group win a recent Kaggle competition (for an overview of their approach, see: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st… I've played around with this a bit over the last few weeks, and have a Matlab implementation publicly available from my Github at: https://github.com/Philip-Bachman/NN-Dropout.

Glad you asked... Definitely image recognition: http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf Speech recognition: http://www.cs.toronto.edu/~hinton/absps/RNN13.pdf Natural language processing: http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutor… http://aclweb.org/anthology/N/N13/N13-1090.pdf If you're into kaggle competitions: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st… I don't think there are going to be any further major advances in eg SVMs or random forests (famous last words maybe...) Neural nets, on the other hand, are just scratching the surface of what's possible. So right now they are state of the art in some historically very difficult areas. But these are early days still.