Datawocky

We could certainly rattle off more terms :) Kernel density estimation? Deep learning? Semi-supervised learning? But it's amazing how far you can get with some simple scatterplots, histograms and linear/logistic regressions. a) Most of the time, more data beats better algorithms: http://anand.typepad.com/datawocky/2008/03/more-data-usual.html b) Even those seemingly simple things get complicated when you have a lot of data: http://komarix.org/ac/lr

I find Netflix's recommendations to be far superior. Participants in the Netflix Prize have found that content-based filtering (such as "nano categories") actually hurts relevancy (in RMSE terms) when working with large data sets. That is, when you have a copious amount of data, it is usually better to let the data speak for itself. Similarly, it has been noted by Google and others that the better your data, the less impressive your algorithms need to be. Most people think Google's success is mainly attributed to magic algorithms, but most of their effort of late has been in increasing the quality and breadth of their data. http://anand.typepad.com/datawocky/2008/03/more-data-usual.html

This line of thought follows previous posts on the same subject. It's very Norvig-esq (http://www.youtube.com/watch?v=LNjJTgXujno), but there's also http://anand.typepad.com/datawocky/2008/03/more-data-usual.html and also Chris Anderson and Wired's flame bait http://www.wired.com/science/discoveries/magazine/16-07/pb_theory (that month's wired was dedicated to this subject) And like someone at the previous discussions has said, this is the base of the scientific method, not it's death

I'm currently pursuing this course on Coursera. We are currently only halfway through the course but I think I can share a few thoughts on the course. Pros - Faculty: Like most MOOCs, MMDS is taught by one of the best faculty from the field. I've been an avid follower of Anand Rajaraman's blog [0] before I joined this course and I have to say the enthusiasm of the faculty is infectious and their expertise with the material is markedly evident. - Difficulty: MMDS is a CS graduate level course (CS246) from Stanford. That means the topics are not trivial, the lectures are dense and you as a student are expected to invest significant time into understanding the material. Since this is hard, grasping the concepts and getting the quiz right is quite gratifying. Few lectures from every week are tagged as advanced and students who view and answer all advanced questions get a certificate of distinction [ not quite relevant but might provide the necessary incentive / motivation to a few students] - Material: The syllabus and the topics covered in this blog are extremely relevant for any one aspiring to work in the data mining / machine learning field. Having done Andrew Ng's ML course, this course acts a perfect supplement and covers a lot of practical aspects of implementing the algorithms when applied to massive data sets. For example, a recent lecture talked about how the BFR algorithm[1] for finding clusters works better than k-means for a very large dataset. - Book: The…

This article seems to be discussing a very different 'Big Data' to that I see. I'm sure the author's points are true in their world, but largely they aren't in mine (mostly online media stuff). Here's a rebuttal of a sort: This data collection is a recent phenomenon: only since media moved online has it been cheap enough to collect sufficient data to do interesting things. Furthermore, you can actually use that data is more meaningful ways (e.g. recommendation systems) that weren't possible before. That value is not being extracted by the companies that do currently collect data: surprisingly true from my discussions with media companies. The largest companies (Yahoo!, Netflix, etc.) sure do, but the second tier don't. That companies will need outside help to extract insights: This perhaps indicative of the biggest difference between our worlds. It seems the author is talking about batch jobs, where you do some rudimentary analysis on a data, makes some slides, present to management, and go home. The people I know are talking about deploying automated systems into production (think recommendation systems again). This requires a large array of skills (machine learning, software engineering, distributed systems, etc.) that many companies don't have. "Minitab, SAS, SPSS, Systat" are not sufficient tools in this world. The insights gathered from ever larger data sets have more value and are more accurate than insights gathered from smaller data sets: The conventional wisdom…

"Are Machine-Learned Models Prone to Catastrophic Errors?" http://anand.typepad.com/datawocky/2008/05/are-human-experts-less-pron…

Bingo. More data beats better algorithms, see http://anand.typepad.com/datawocky/2008/03/more-data-usual.html

Sometimes more data can beat better algorithms: http://anand.typepad.com/datawocky/2008/03/more-data-usual.html

Mountain View/ San Bruno, CA I work for company called @WalmartLabs. It was previously known as Kosmix and acquired very recently by Walmart. I joined almost 9 months ago, and really loving it so far. We have very fun and social work environment. I like the work we are doing here. You can find more about the vision of the company from Anand's (our Founder) blog http://anand.typepad.com/datawocky/2011/04/retail-social-mobile-walmar… And yes we are looking for more team members who are fun, smart, and share our excitement. You can reach me via [email protected] , and we do H1B as well.

Anand Rajaraman's blog post explaining this acquisition: Retail + Social + Mobile=@WalmartLabs. http://anand.typepad.com/datawocky/2011/04/retail-social-mobile-walmar…

A better model: Perhaps smaller democracies? Instead of everyone having 1/300,000,000-say in 300,000,000 lives, let them have bigger chunk in smaller systems. Stronger, faster and more direct feedback can enhance control loops without adding more sophistication otherwise. More data usually beats better algorithms. (http://anand.typepad.com/datawocky/2008/03/more-data-usual.html)

Actually, scaling twitter is a solved problem. There is a company in India called SMS GupShup that does the same thing as twitter, except they have 3 times the number of users and more SMS users. And they don't have downtime. http://anand.typepad.com/datawocky/2008/06/indias-sms-gupshup-has-3x-t…

Yes, a week ago http://anand.typepad.com/datawocky/2008/06/angel-vc-or-bootstrap.html

How about this? http://anand.typepad.com/datawocky/2008/03/more-data-usual.html Norvigs point was similar, but Anand has more convincing examples.

Really? Can you say why? I haven't tried it, but link counts look like a reasonable first approximation of pagerank. I thought this is also the consensus among researchers, e.g.: http://anand.typepad.com/datawocky/2008/03/more-data-usual.html http://research.microsoft.com/users/nickcr/pubs/craswell_trec02.pdf

Check out our tool, http://tagger.flaptor.com It's based on a Bayesian algorithm plus a bunch of other heuristics for fine-tuning. In our case, the classification algorithm is not nearly as important as how we select documents for the training set. Also, take a look at this post that was mentioned here a few days ago: More data usually beats better algorithms: http://anand.typepad.com/datawocky/2008/03/more-data-usual.html

PageRank and RankBrain both seem to be (complicated) features to the actual core search algorithm, which is some unspecified machine learned model [1]. PageRank, for example, can't be the whole search algorithm since it's not even query-dependent. It would just put the same most authoritative document at the top for every query. Similiarly, RankBrain doesn't sound like it could be the whole algorithm. It sounds like it is just a text understanding model (which wouldn't know anything about e.g. the global reputation of the document, the popularity of the document, etc.). In fact, the article explicitly confirms this: >RankBrain has become the third-most important signal contributing to the result of a search query, he said. I'd guess some kind of composite content quality signal and some kind of composite popularity signal sit at #1 and #2. [1] - I don't have any insider knowledge about how Google works, but this article suggests that they were getting ready to switch from a hand-tuned model to a machine learned one in 2008: http://anand.typepad.com/datawocky/2008/05/are-human-experts-less-pron…

Hey, thanks for taking the time to reply. I realize that my comment probably comes across as being quite personal, but it more reflects my frustration at the lack of insight into the Netflix Prize that exists in the blogosphere. Here are some prime examples of people making lots of noise without any data or science to back it up: http://anand.typepad.com/datawocky/2008/03/more-data-usual.html http://scienceblogs.com/cortex/2009/08/netflix.php There's actually very few people who have made genuine contributions toward winning the Netflix prize, as can be see in the winning team's final publications. They only list about a half-dozen key papers as references. Anyways, I apologize for directing my comments specifically at you. I totally agree with your basic point, and this is a problem I've been spending a lot of time thinking about myself. My personal view is that explicit rating systems should be totally eliminated, in favor of using data gathered automatically, without asking the user to provide a subjective rating. I don't either have any evidence to prove that's better, mind you. :)