Hunch.net

Do note that the more the gate is kept, the stronger the incentive for cheating. Moreover, peer review is overrated. An experiment by the NIPS conference in 2014 found that ~60% of submissions are in the "gray middle" - not obviously great or obviously crap. In more detail: they split the PC in two, and had 10% of papers (166) reviewed by both halves. Each half had to accept ~37 papers. The halves disagreed on 21 accepted papers [1]. So yeah, if your submission is sciencey, it's the flip of a coin whether peer review will accept it. [1] https://hunch.net/?p=467864

I love reading these two for Python/Recommender Systems: 1. Ben Frederickson: https://www.benfrederickson.com/blog/ 2. Erik Bernhardsson: https://erikbern.com/ And for pure ML: 1. Ferenc Huszár: https://www.inference.vc/ 2. John Langford: https://hunch.net (although he hasn't written a technical post in a while)

I can recommend a few great podcasts: http://www.thetalkingmachines.com/blog/ http://partiallyderivative.com/ Also http://hunch.net/ and https://www.quora.com/What-are-the-best-machine-learning-blogs-or-reso… but there are sooo many. i would say decide what parts of ML you like and focus on those first...if not it's an overwhelming rabbit hole of things to learn

I can totally appreciate the value of having well defined and well understood definitions and notations that might require lots of background to understand. This allows for precise and concise discourse. But there have been many published bounds in Machine Learning Theory that turned out to be completely vacuous. That is, they prove a bound on generalization error that is always greater than 1. http://hunch.net/?p=181 To me, that definitely seems like complicated math for the sake of showing off and writing papers rather than writing for clarity and demonstrating something useful.

You're not alone. I don't know if you have a ML background, but it has been somewhat widely discussed in the community that the way Netflix scored the competition -- RMSE -- isn't the best way. E.g. http://hunch.net/?p=949 and http://andrewgelman.com/2008/11/netflix_prize_s/ In the OPs article they mention they monitor if a movie is watched to completion, which gives them a much better metric to optimise. The other issue is that this is really a sequential decision making problem. Recommending a movie has an opportunity cost -- there are other movies you don't recommend -- and the recommendation is an ongoing process, so it is probably best to spend some time exploring the user's taste on the assumption this will let you make better recommendations in the future. Accounting for these issues is much harder in a competition format.

> AIXI is a pretty deeply flawed theory of "universal intelligence" What would you propose as an alternative? If nothing, fine, but how can we relate, when we only have a single best (or if you prefer: flawed) thing? > It can't handle modelling itself It can add its internal states to the environment, and hence model these internal states. > the large dimensionality of its hypothesis space Its bounded by how many compressors / programs are available to it. Calculating the length of these programs that are consistent with the environment is feasible. > handle imprecise stimulus information I'm not sure if you mean imprecise stimuli here or imprecise sensors. > can be made arbitrarily "stupid" This can be seen as a flaw, or as a simple property (or even a feature). Something that can be optimal, and "stupid", does not detract much from its ability to be optimal. > before we get to mere computational limitations or the uncomputability issue Just because it is uncomputable, does not mean the theory is flawed. Sure, it is not practical, and we like practical things, but it is still valuable to have such a theory. Especially when approximations do yield practical applications. Marcus Hutter on these issues: http://hunch.net/?cat=14

Best of luck. Do you have a hypothesis? More experienced ML folks don't seem to be on board [1] [2]. I haven't seen anyone posit how to build a machine that can set its own goals. I'm sure you'll build and learn something cool nonetheless Another organization that tried to build AI was called Thinking Machines [3]. They were good at parallel programming. A few of them would later form "Ab Initio Software" [4]. Their motto is "from first principles," which Elon likes to say. Socrates did too. My first job out of college trained us on that software. It's easy to use even if you're not a programmer, which was impressive for parallel software in 2004. I still get job emails for having it on my resume. The company is private and does not share its training resources. Boston mentality, I guess. Despite being so closed, they appeared to be quite successful in 2004 and have managed to stick around. [1] http://hunch.net/?p=3692542 [2] https://www.facebook.com/yann.lecun/posts/10153442884182143?__mref=mes… [3] https://www.technologyreview.com/s/406781/thinking-machines/ [4] http://www.abinitio.com/

Preface: AlphaGo is an amazing achievement and does show an interesting advancement in the field. Yet ... it really doesn't mean almost anything that people are predicting it to mean. Slashdot went so far as to say that "We know now that we don't need any big new breakthroughs to get to true AI". The field of ML/AI is in a fight where people want more science fiction than scientific reality. Science fiction is sexy, sells well, and doesn't require the specifics. Some of the limitations preventing AlphaGo from being general: + Monte Carlo tree search (MCTS) is really effective at Go but not applicable to many other domains we care about. If your problem is in terms of {state, action} pairs and you're able to run simulations to predict outcomes, great, but otherwise, not so much. Go also has the advantage of perfect information (you know the full state of the board) and deterministic simulation (you know with certainty what the state is after action A). + The neural networks (NN) were bootstrapped by predicting the next moves in more matches than any individual human has ever seen, let alone played. It then played more against itself (cool!) to improve - but it didn't learn that from scratch. They're aiming to learn this step without the human database but it'll still be very different (read: inefficient) compared to the type of learning a human does. + The hardware requirements were stunning (280 GPUs and 1920 CPUs for the largest variant) and were an integral part to…

Readers who enjoy banditry may also enjoy John Langford's http://hunch.net

This is a repost. The original link should be: http://hunch.net/?p=3692542 . Other discussion from Yann Lecun https://www.facebook.com/yann.lecun/posts/10153426027052143 and http://www.information-age.com/technology/applications-and-development…

I'm not sure if it's a deep learning framework. Perhaps they are in some ways complementary http://hunch.net/?p=2875548. Who knows, maybe CNTK is one big reduction.

It is hard to emphasize how bad overfitting on the test set is. Your cheating analogy is accurate, and cheating in science is a serious matter. If it is unintentional, you should have your work rejected or retracted. If it is intentional, you should have your work rejected or retracted and, frankly speaking, you should most likely leave the field. Science is based on a huge deal of trust and violating that trust for your own short term gains is inexcusable. Even worse, any honest scientist will have find it more difficult to improve upon results that can be explained by overfitting. A good short description on several issues that plague the field is "Clever Methods of Overfitting". http://hunch.net/?p=22

Metacademy ( http://www.metacademy.org/ ) is exactly what you described. The website gives a concept map and each concept has links to free educational resources. Here's the concept map for SVD: http://www.metacademy.org/graphs/concepts/singular_value_decomposition… You can read a bit about the ideas behind Metacademy here: http://hunch.net/?p=2714

Co-creator of Metacademy here. Colorado and I are intending Metacademy to be a “package manager for knowledge,” where you can easily find the particular thing you want to learn about (e.g. deep neural nets) without having to track down all the prerequisites (e.g. gradient descent) yourself. Basically, we’ve annotated a dependency graph for the core concepts in the field, and it uses the graph to produce step-by-step learning plans geared directly towards your goals. Sorry if you can't reach the site. We’re both doing this as volunteers, and I guess we weren’t prepared for the level of traffic we’re getting. In the meantime, you can find more details about our high-level goals here: http://hunch.net/?p=2714 If you don’t have a particular goal in mind and just want a general overview, check out the roadmaps, e.g. http://www.metacademy.org/roadmaps/rgrosse/bayesian_machine_learning

Agree. I found this post by John Langford more informative, on a similar vein: http://hunch.net/?p=2562

Unknown is relative, but here are some of the rss feeds I'm subscribed too: http://nosql.mypopescu.com/ http://lambda-the-ultimate.org/ http://eli.thegreenplace.net/ http://www.stefankrause.net/wp/ http://mechanical-sympathy.blogspot.com/ http://www.cowtowncoder.com/blog/blog.html http://highlyscalable.wordpress.com/ http://hunch.net/ http://perspectives.mvdirona.com/ http://www.scottaaronson.com/blog/ http://www.softwarebyrob.com/ http://blog.mikemccandless.com/

I'll ask him whether he considers himself mostly a pencil pushing fraud when I see him in December. I'll also see what he thinks of the claim that VW's usefulness is "not due to any theory". I'll think you'll be surprised given he's written a blog post on this topic titled "Use of Learning Theory" http://hunch.net/?p=496 at his blog which, by the way, is called "Machine Learning (Theory)". It's easy to dismiss what you don't understand but you should consider the possibility that it is significantly more difficult develop algorithms like those in VW and "bang out" implementations of them that are fast and correct.

This is the most important critique of A/B testing. It far outweighs the traditional hoopla about simultaneous inference and Bonferonni corrections. Epsilon greedy does well on k-armed bandit problems, but in most applications you likely can do significantly better by customizing the strategy to individual users. That's a contextual bandit and there are simple strategies that to pretty well here too. For instance: http://hunch.net/?p=298 http://hunch.net/~exploration_learning/main.pdf http://web.mit.edu/hauser/www/Papers/Hauser_Urban_Liberali_Braun_Websi…

John Langford's comments on the subject: http://hunch.net/?p=2341

Vowpal Wabbit (one of the fastest learning implementations, from John Langford, also of Yahoo!) now has an "online" LDA implementation (Hoffman et al 2010: http://www.cs.princeton.edu/~blei/papers/HoffmanBleiBach2010b.pdf). It's probably the fastest single-core LDA implementation. http://hunch.net/?p=1594 http://www.machinedlearnings.com/2010/12/lightning-fast-lda.html

I would wager most of the people commenting don't work in search and how to use datamining to improve search. I think it is useful to see the perspective of someone that actually thinks long and hard about building these types of systems before commenting on whether it is fair or innovative or just plain stealing. http://hunch.net/?p=1660

MLComp (http://mlcomp.org) is a site for comparing different algorithms outputs over various datasets. The site appears to be down right now, but you can go here (http://hunch.net/?p=1309) for an overview of the site. (If it comes back up) It might help you get a feel for some real differences in learning mechanisms.

Yehuda later wrote http://glinden.blogspot.com/2008/03/using-imdb-data-for-netflix-prize.… and http://hunch.net/?p=331 that using movie metadata has produced no measurable improvement in RMSE.

The machine learning blog at http://hunch.net/ is a good resource.

If you're looking for a big data cruncher on the cheap, I think your choice of programming language becomes important. Ditch Python for something a little snappier. There's a discussion of languages for Machine Learning at http://hunch.net/?p=230.

To use the test set explicitly for evaluation is a deadly sin. When found out, you'd face serious damage to your reputation (Like Baidu did a few years ago). [1] What the decreasing performance results on the remade CIFAR-10 test set shows, is probably more akin to a subtle form of overfitting (Due to these datasets being around for a long time, the good results get published, and the bad results discarded, leading to a feedback loop). [2] It is also possible the original test set was closer in distribution to the train set, than the remade one. The ranks stay too consistent for test set evaluation cheating. I also think the "do not trust saliency maps" is too strongly worded. The authors of that paper used adversarial techniques to change the saliency maps. Not just random noise or slight variation, but carefully crafted noise to attack saliency feature importance maps. > For example, while it would be nice to have a CNN identify a spot on an MRI image as a malignant cancer-causing tumor, these results should not be trusted if they are based on fragile interpretation methods. Interpretation methods are as fragile as the deep learning model itself, which is susceptible to adversarial images too. If you allow for scenario's with adversarial images, not only should you not trust the interpretation methods, but also the predictions themselves, destroying any pragmatic value left. It is hard to imagine a realistic threat scenario where MRI's are altered by an adversary,…

Academia is basically "self-reporting on the honor system". It works generally but there are lots of holes. Ultimately, "trust but verify" is necessary to avoid getting caught in a wave of hype, or at least having someone in your own "circle of trust" say it works. This system leads naturally to elitism and a bunch of other problems which are seen in academia, but it seems better than the current alternatives to me given the current rabid focus on exact percentage score instead of quality/utility of an idea. The "right way" to do it is test once only per model/paper. If you are interested there are a huge number of sneaky ways overfitting can happen in ML [1]. Also interesting that you too see crypto and ML as related - I see them as opposites of the same coin. One tries to pull signal out of noise, the other tries to bury the signal in noise... but special noise. [1] http://hunch.net/?p=22

A few others: Machine Learning (Theory), http://hunch.net/, is a good place to learn about new machine-learning papers. It doesn't really do research exposition on the blog, but it posts about recent ML conferences, highlighting some of the papers the author finds interesting. Embedded in Academia, http://blog.regehr.org/, is about 50% personal stuff, but 50% posts on John Regehr's work on C-compiler fuzzing, with some interesting examples if you're into compilers or the finer points of C semantics. Proper Fixation, http://www.yosefk.com/blog/, is by an embedded developer (not academic), and not always about research, but it has some good researchy and expository posts. For example, it has the best concise overview I've found of how SIMT/SIMD/SMT relate (http://www.yosefk.com/blog/simd-simt-smt-parallelism-in-nvidia-gpus.ht… While it's a mathematics blog, Terence Tao's blog, http://terrytao.wordpress.com/, has a lot of content likely of interest to computer scientists as well. In particular, his blog-exposition versions of papers are often a better introduction to recent research for nonspecialists than anything in the official published literature is. Tomasz Malisiewicz's computer-vision blog, http://quantombone.blogspot.com/, has intermittent but often quite good posts on object recognition and similar topics. Of course I can't refrain from mentioning my own quasi-blog, http://www.kmjn.org/notes/, though only about 1/4 of it is on computer science (about 4/5 of my…

Yes, it is the case here that a bayesian approach can lead to a previous empirical formula falling out. What I was saying as well is that this regularization + test set approach is also valid (and sometimes slightly more or less general than the bayesian approach, since, for example, SVMs fall naturally out of thinking about regularization but they have no analogue in bayesian classifiers). It also goes the other way, and some formulas are first proposed in a more bayesian-ish context and then extended to some simpler-looking empirical formulas (for example, the jumps from hidden Markov models to max-ent Markov models to conditional random fields to max-margin Markov networks to structured SVMs). There are more approaches to machine learning, and in John Langford's blog there is a nice table showing the merits and flaws of many of them: http://hunch.net/?p=224 . But you must keep in mind that you can find many equivalencies between these approaches (boosting for example can be seen as a loss minimization with regularization, and max-ent can be seen as a special case of a bayesian model, etc).

And from a fellow researcher: http://hunch.net/?p=1172