Mat Kelcey

have a look at Simhash as well: https://www.matpalm.com/resemblance/simhash/

Downvoters, please see http://matpalm.com/resemblance/simhash/ https://en.wikipedia.org/wiki/SimHash Simhash, a type of local sensitive hashing - using hash functions on ngrammed data. That is exactly what Vowpal Wabbit does.

Source: http://matpalm.com/blog/counting_bees/ Once again, mods don't replace the link to an obvious blog spam article just because it's from TechCrunch.

Previous discussion: https://news.ycombinator.com/item?id=17103635 The blog post the article is actually about: http://matpalm.com/blog/counting_bees/

Something not entirely quite unlike this http://matpalm.com/blog/2011/08/13/wikipedia-philosophy/

For fast and larger scale near-duplicate detection see Google All-Pairs Similarity Search: https://code.google.com/p/google-all-pairs-similarity-search/ . The SimHash algo: http://matpalm.com/resemblance/simhash/ and Gensim's Similarities Class: http://radimrehurek.com/gensim/similarities/docsim.html For blogspam detection can't PG whip up a nice Bayesian model for that ;) ? Again if you want fast and large scale spam filters see Vowpal Wabbit: http://hunch.net/~vw/ . For something more advanced see scikit-learn: http://scikit-learn.org/ which has many algo's up for this task. A new option you may not have considered yet, is to crowdsource this task. Just offer a labelled dataset and let the HN and ML community have a go at it. I'd love to do content and URL-based spam modelling.

Cool! @matpalm, please run your English extractor scripts on it! (http://matpalm.com/blog/2011/12/10/common_crawl_visible_text/) 'If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.' This is annoying, that they're using the enterprise sales model for distribution. Just put it on S3.

This blog post: http://matpalm.com/blog/2012/01/01/common_crawl_collocations/ mentions that commoncrawl's data is last updated September 2010. Does anyone know if that's still the case?

The author pointed this out; see "optimizing even more" under http://matpalm.com/median/erlang_multi.html

The passage quoted above refers to the way google plans to audit FLoC, not FLoC itself. Each browser is assigned a single floc ID, which is (for now) a 50-bit hash that is not human readable on its own. But the floc ID is intended to capture something meaningful about browsing behavior. Two users with similar browsing history are likely to have similar, if not the same, floc IDs. Floc IDs which are close together (hamming distance-wise) also reflect similar browsing histories. (http://matpalm.com/resemblance/simhash/) It will be totally possible to gather data about what kind of people belong to a particular floc -- provided that you operate a large website or ad network. It will probably not be possible for normal people. There's a risk that advertisers will learn that a particular floc ID corresponds to a "sensitive category" of users. That's what google is trying to prevent. Google plans to audit floc IDs for correlation with visits to sensitive websites. In order to do so, it's assigning a "sensitivity" label to certain websites, and then applying that label to each user who visits those websites. Then it will run analyses to see whether users with a particular floc ID are more likely to have a particular label. Floc IDs that correlate too strongly with a particluar label will be thrown out. The passage is attempting to explain why this does not prevent adtech from learning other meaningful things about flocs -- for example, that users in a specific floc are…

It does introduce new information. This is how semi-supervised learning works. Many words may be missed by the weak model, but can be inferred correctly from their context. Then you have new labels to train the weak model on, to make it better at those words. The way I'm describing it is probably not the optimal way to do it. I don't know if there is a better way. But the point is it must be possible to take advantage of the vast quantities of unlabelled data we have. Humans brains somehow do something similar. Semi supervised learning is really cool. I saw one successful example where they labelled just a few emails as spam and not spam. Then they trained used their weak classifier to label thousands of unclassified emails, and then used those as training data for an even stronger model. It actually works: http://matpalm.com/semi_supervised_naive_bayes/semi_supervised_bayes.h… https://en.wikipedia.org/wiki/Semi-supervised_learning