Mike McCandless

It has long since been cleaned up, but the history of Lucene's Levenshtein automatons are a fascinating mix of pragmatism, hackery, grit, and engineering. Using a Python library to generate crazy-looking Java code, which you can't (yet) understand, but whose tests pass and benchmarks look amazing. - https://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-time… - https://www.youtube.com/watch?v=4AbcvHKX8Ow (Levenshtein distances are used as a measure between fuzzy/misspelled matches, and was ironically misspelled initially - https://lucene.apache.org/core/7_4_0/suggest/org/apache/lucene/search/…

The Lucene search library has something like this. Data is stored /retrieved via codecs. Most write binary data, but one codec actually writes out all data as text. It is, of course, too slow for production use, but it is great for figuring out what is going on under the covers. https://blog.mikemccandless.com/2010/10/lucenes-simpletext-codec.html?…

A footnote: For Java devs out there, the Lucene codebase is a joy to read [0]. Esp, the APIs. Highly recommend it just for the documentation which is top-notch. Michael McCandless, Lucene's committer-in-chief, sometimes blogs about its internals [1]. [0] https://github.com/apache/lucene-solr/tree/master/lucene [1] http://blog.mikemccandless.com/2019/10/concurrent-query-execution-in-a…

People with an interest may want to check out Azul's Zing JVM and the following blog post testing it out with Lucene... http://blog.mikemccandless.com/2012/07/lucene-index-in-ram-with-azuls-…

Unknown is relative, but here are some of the rss feeds I'm subscribed too: http://nosql.mypopescu.com/ http://lambda-the-ultimate.org/ http://eli.thegreenplace.net/ http://www.stefankrause.net/wp/ http://mechanical-sympathy.blogspot.com/ http://www.cowtowncoder.com/blog/blog.html http://highlyscalable.wordpress.com/ http://hunch.net/ http://perspectives.mvdirona.com/ http://www.scottaaronson.com/blog/ http://www.softwarebyrob.com/ http://blog.mikemccandless.com/

It's OK to not like Solr. At the same time, half of the features I listed above, are quite new (SolrCloud, docValues, LTR, Config and Schema API, JSON support, etc). And the other half - the 'old' part you may not like - are battle-tested, multiple-times speed-optimized pieces of code. Like this one: http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times… Moreover, their architecture makes very clear that they are making very similar choices, it is just their implementation is much fresher. Sure, there is crud in Solr, it is an open-source product driven by the user needs. Sure, it is possible that - for some usecases - Java is disadvantage. I would have loved that refreshed comparison to be in the article. It is very jarring that it was not. As it is, it felt that they walked away from 2015 and have not looked since. Even though their "simpler" approach did not work out and they had to throw it away.

Lucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts with C++ and saw massive gains (but with a disclaimer that this was in no way indicative that Java wasn't fast). I've considered trying to implement something like Lucene in, say, Rust, but then I see just how utterly massive Lucene is. Just the fuzzy search part alone required implementing code to generate code from a Russian PhD thesis they didn't fully understand.[1] So, no matter how many cycles the JVM is needlessly burning, Lucene just seems to advanced to write it without the overhead. (And maybe my intuition is just wrong and the overhead is only a few percent.) 1: http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times…

There's a quite amusing blog post about implementing the automaton for Lucene: http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times… As an aside, I know one of the authors of the original paper personally. Quite interesting guy, and the remark about the paper being "nearly unintelligible" is fitting.

Jason from Elastic here. A shard can handle concurrent writes from clients, there are multiple write threads scaling to the number of logical processors exposed to Elasticsearch, and we have invested in making Lucene accommodate this concurrency[0][1][2]. [0]: http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucen… [1]: https://issues.apache.org/jira/browse/LUCENE-3023 [2]: http://blog.mikemccandless.com/2017/07/lucene-gets-concurrent-deletes-… Disclaimer: I am an engineer on the Elasticsearch team; I welcome any and all feedback.

If anyone wants to know how deep down the rabbit hole this stuff goes, they should read this blog post [1] on writing a Levenshtein Automaton to speed up fuzzy matching in Lucene by 100 times. It gets deep! [1]: http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times…

To add another one: Lucene uses a finite state transducer to represent a part of its term index: http://blog.mikemccandless.com/2010/12/using-finite-state-transducers-… --- It's essentially a dictionary as you say, but instead of a direct NLP use case, it might map a word to some numerical value (like a file offset where the term's postings list resides). I actually haven't been able to find anyone else using transducers for this purpose (sans OpenSextant, I think), but I'd be curious to hear about it if others knew. It seems like a really awesome way to represent nearly arbitrary maps/sets of billions of strings.

Something of a story at http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times…

Also interesting, the cutting edge work lucene has done to make this possible in 4.x, using universal automatons to speed up fuzzy, regex, and other searches. http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times…

Mike McCandless gives a good overview here: http://blog.mikemccandless.com/2012/07/lucene-400-alpha-at-long-last.h… He has some other posts on there that do a deeper dive as well.

This blog post [http://blog.mikemccandless.com/2012/08/lucenes-new-blockpostingsformat… and the associated Lucene Jira issue [https://issues.apache.org/jira/browse/LUCENE-3892] may be of interest to those who are excited by this paper. If I'm reading it right, I think the main conclusion from the Lucene patch was that using a straight "FOR" (Frame of Reference) approach was faster in the real world than using "PFOR" (Patched Frame of Reference), and only slightly worse in index size. I haven't read this new paper yet, but would be interesting to compare and contrast the approaches.

I believe we're somewhat constrained by our license with Oracle regarding what we can publish for benchmarks. But others are not, and this piece by Mike McCandless gives some good insight into what's possible with Zing: http://blog.mikemccandless.com/2012/07/lucene-index-in-ram-with-azuls-…