RARE Technologies Blog

I run a company that specializes in design & implementation of kick-ass ML solutions [1]. We've had successful projects in quite a few industries at this point: LEGAL INDUSTRY Aka e-discovery [2]: produce digital documents in legal proceedings. What was special: stringent requirements on statistical robustness! (the opposing party can challenge your process in court -- everything about way you build your datasets or measure the production recall the has to be absolutely bullet proof) IT & SECURITY Anomaly detection in system usage patterns (with features like process load, frequency, volume) using NNs. What was special: extra features from document content (type of document being accessed, topic modeling, classification). MEDIA Built tiered IAB classification [3] for magazine and newspaper articles. Built a topic modeling system to automatically discover themes in large document collections (articles, tweets), to replace manual taxonomies and tagging, for consistent KPI tracking. What was special: massive data volumes, real-time processing. REAL ESTATE Built a recommendation engine that automatically assembles newsletters, and learns user preferences from their feedback (newsletter clicks), using multi-arm bandits. What was special: exploration / exploitation tradeoff from implicit and explicit feedback. Topic modeling to get relevant features. LIBRARY DISCOVERY Built a search engine (which is called "discovery" in this industry), based on Elasticsearch. What…

If you actually want people to subscribe to your podcast and listen to it, you need to give them what they need to do that. Put the words "RSS" or "feed" on the page with a link to the feed. Don't make me view the page source to find it, which is what I had to do. https://rare-technologies.com/comments/feed/

I don't think you are correct here. The advantage of using sparse storage and sparse matrix multiplication is that you can get savings in both storage and runtime. There would be no point in using sparse storage if runtime still scaled with the cube of vocab size. It would be that way if the best way of getting sparse SVD is by materializing a dense matrix product, but people have discovered smarter ways using sparse matvec. The time complexity for obtaining k eigenvalues seems to be O(dkn) where d is the average number of nonzeroes per row, and n is the vocab size [1]. Therefore, one can assert that sparse SVD too is linear in vocab size just like word2vec. This is corroborated by the link elsewhere in this thread that shows SVD enjoying lower wall clock time on the 1.9B word Wikipedia dataset [2]. [1] https://en.wikipedia.org/wiki/Lanczos_algorithm#Application_to_the_eig… [2] https://rare-technologies.com/making-sense-of-word2vec/

> However, I think this just calls out all the open questions about this style of consulting and doesn't answer them. How does one get clients, how does one keep a steady income, how to invest for retirement, how does one convince clients you have the right skills, how do you know you have the rights skills, etc.? He touches on some of these issues in a podcast interview with Radim Rehurek (of Gensim fame). https://rare-technologies.com/rrp-2-john-d-cook-consulting/

This is (2017), specifically 2017-10-28. Also, it's Part 1 of 2. Part 2 of 2 is here: https://rare-technologies.com/sharing-facets-dive-visualizations/ and is also (2017), specifically 2017-11-11.

This direct link works for me -- https://rare-technologies.com/machine-learning-hardware-benchmarks/

For this kind of thing I've heard that Annoy is pretty good. No personal experience though: https://pypi.python.org/pypi/annoy Related shootout: https://rare-technologies.com/performance-shootout-of-nearest-neighbou…

Blender Python lib by default is not optimized much. It has nothing to do with Python as a language. Use numpy for matrices. If you have to implement an algo with a hot inner loop, use cython or numba. I've never seen 100x difference in Python-C++ rewrite if Python was optimized already. Here is a good article about some of the options: https://rare-technologies.com/word2vec-in-python-part-two-optimizing/

Here is an interesting comparison between FastText and word2vec. https://rare-technologies.com/fasttext-and-gensim-word-embeddings/

Have you tried http://www.onelook.com/reverse-dictionary.shtml? Edit: There is also : http://wordgrabbag.com/ - which seems to do this better, but I'd probably use a combination of those. Edit: Just remembered that this tutorial loads all of the word2vec data (and at the bottom) allows API access into it for similar words. http://rare-technologies.com/word2vec-tutorial/

Quoc V. Le also helped develop Word2Vec, released as opensource couple of years ago: https://code.google.com/p/word2vec/ You can play with it here: http://rare-technologies.com/word2vec-tutorial/ (a bit further down) Google seems rather heavily invested in NLP, can't blame them as it's a very exciting domain to explore.

This isn't a real downstream task, but one of the researchers at RaRe compared FastText to word2vec/gensim/skipgram word embeddings on the original testsets for the 'semantic' and 'syntactic' analogy tasks from the word2vec papers here: https://rare-technologies.com/fasttext-and-gensim-word-embeddings/ The conclusion: These preliminary results seem to indicate fastText embeddings are significantly better than word2vec at encoding syntactic information. This is expected, since most syntactic analogies are morphology based, and the char n-gram approach of fastText takes such information into account. The original word2vec model seems to perform better on semantic tasks, since words in semantic analogies are unrelated to their char n-grams, and the added information from irrelevant char n-grams worsens the embeddings. Personally I think those analogy testsets are not very good, because they just test all pairs of relations between a very small number of words from very limited domains (like capital and country names). One advantage of FastText should be better learning on small amounts of data like Wikipedia.

Like a lot of work in NLP at the moment, this is a reasonably straight-forward mash-up of existing techniques. This particular idea is pretty obvious. What wasn't obvious was whether it would work well, which was why nobody else got around to trying it yet. The experiments are nicely conducted, with strong baselines across multiple evaluations. The authors also include experiments with variations of the idea, to further validate the approach. To understand the technique, first understand word2vec: http://rare-technologies.com/word2vec-tutorial/ http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/ Now understand part-of-speech tagging: http://spacy.io/blog/part-of-speech-POS-tagger-in-python/ By default word2vec gives you clusters for each word, this paper is giving you clusters for word_POS, e.g. The_DT apple_NNP employee_NN is_VBZ eating_VBG an_DT apple_NN. The same trick is done with named entity labels as well. The following papers explain how the new word vectors are used in a dependency parser: Collobert and Weston (2011): http://arxiv.org/pdf/1103.0398.pdf Wang and Manning (2014): http://cs.stanford.edu/~danqi/papers/emnlp2014.pdf Yoav Goldberg (2015): http://u.cs.biu.ac.il/~yogo/nnlp.pdf Survey/review, aimed at grad students