Nick Johnson

Vaguely related technique with similar goals (but I love the one posted here) http://blog.notdot.net/2007/9/Damn-Cool-Algorithms-Part-2-Secure-permu…

I'm a bit confused about the "hash" part of this... wouldn't you want to be able to go back from the hashed string to the original location? I can't think of much of a use for only being able to test locations to see if they match a proximity hash. Internally, the space-filling curves are super interesting. I highly recommend this writeup [1] on how they're used in things like planet-sized maps. I've heard rumors that Google adjusts the offset of their space-filling curves to minimize large discontinuities near land and push them over the ocean instead. [1] - http://blog.notdot.net/2009/11/Damn-Cool-Algorithms-Spatial-indexing-w…

That reminds me of my favorite project of all time, the 7400 FPGA: http://blog.notdot.net/2012/10/Build-your-own-FPGA

This is a great reference that I relied on while struggling with my own implementation: http://blog.notdot.net/2009/11/Damn-Cool-Algorithms-Spatial-indexing-w…

It could see cases where there is an impedance mismatch between the guest FPGA running on the host FPGA. The idea of the genetic algorithm to map out the search space is to find some identity function HDL that allows for a clean 1:1 or 2:1 mapping of resources without incurring some combinatorial inefficiency. Maybe c-slowing could help with the timings, even if the area used was 4x it would still be a great research platform for reconfigurable computing since the bitstreams are proprietary, not documented and not portable. http://blog.notdot.net/2012/10/Build-your-own-FPGA

Nick's blog is a must read for these sort of algorithms. See his original writeup on the HLL here: http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinality-Estimat…

If you store data onto 50 DNA strands, can you always read back all the data from all 50 strands, or does one need to store multiple copies of each in case the sequencer can't "find" a particular strand? If one does need multiple copies, it would seem that this method suffers from the coupon collector's problem [1] (i.e. to collect all 50 strands requires collecting 225 random strands on average), and that the retrieval rate could be improved by using a fountain code [2], which allows each strand to simultaneously encode data at multiple addresses, which would decrease the number of strands required to be sampled to only slightly more than the number of strands worth of data requested. [1] http://en.wikipedia.org/wiki/Coupon_collectors_problem [2] http://blog.notdot.net/2012/01/Damn-Cool-Algorithms-Fountain-Codes

this is all new to me and i'm struggling to find a friendly, simple introduction to the field. does anyone have a good link to, say, an introductory article on fountain codes? i'm starting to get a feel for the idea from reading various wikipedia pages, paper abstracts, powerpoints, etc, but a good "layman's" article would be great... [update] huh. just after posting i started searching for "fountain codes" and found this - http://blog.notdot.net/2012/01/Damn-Cool-Algorithms-Fountain-Codes (previously had been searching for luby, raptor, etc).

Agreed re: the quality of Frank McSherry's posts - they're all great fun and push the technical envelope! I'm glad you brought up Hilbert ordering. It's fascinating but I rarely see it. I usually see Z-order curves in their place - simpler and more intuitive but also less effective. Does anyone know of large scale databases with Hilbert ordering out of the box? For anyone who hasn't seen them before, or why they help in ordering, I highly recommend "Portrait of the Hilbert curve"[1] and "Spatial indexing with Quadtrees and Hilbert Curves"[2]. The tldr is that they enable efficient multi-dimensional indexing. Given a request like "X < 40 and Y > 80", most databases need to run "X < 40" and then filter over the results for "Y > 80" - Hilbert curves allow you to find the query's intersection and only then perform minor filtering. As far as large scale use of Hilbert ordering, Google use it internally for much of their geographical work. They even open sourced a Java library specialised in multi-dimensional indexing based on Hilbert curves called Uzaygezen[3]. [1]: http://corte.si/%2Fposts/code/hilbert/portrait/index.html [2]: http://blog.notdot.net/2009/11/Damn-Cool-Algorithms-Spatial-indexing-w… [3]: http://google-opensource.blogspot.com/2008/08/uzaygezen-multi-dimensio…

Nick Johnson's blog about damn cool algorithms http://blog.notdot.net/tag/damn-cool-algorithms

The BK-Tree, which allows fast querying of "close" matches, such as Hamming distance (number of bits different). http://blog.notdot.net/2007/4/Damn-Cool-Algorithms-Part-1-BK-Trees I wrote a Python library implementing them a number of years ago: https://github.com/benhoyt/pybktree

There's a searching concept known as edit (levenshtein) distance that is close but not exactly what you're talking about. To rephrase your question: is there a datastructure that can pre-compute the levenshtein distance for a set or results? Yes, and it's crazy complicated. See Levenshtein Automata[1]. In reality, I would let the professionals handle this by either using Open/ElasticSearch or Apache Lucene. [1]http://blog.notdot.net/2010/07/Damn-Cool-Algorithms-Levenshtein-Automa…

This article is old but really good at explaining HyperLogLog: http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinality-Estimat…

There is a very clever and beautiful algorithm that only uses the xor operation. It's the Luby Transform code, the simplest implementation of a Fountain Code. A Fountain Code lets you transmit a file over an unreliable connection (packets can be unordered, and even lost), with just a unidirectional communication. And this can be done just by xoring a random number of blocks from the file over and over! A very good explanation of the algorithm: http://blog.notdot.net/2012/01/Damn-Cool-Algorithms-Fountain-Codes

There are a couple: http://blog.notdot.net/2012/10/Build-your-own-FPGA https://news.ycombinator.com/item?id=24252737 https://github.com/haojunliu/OpenFPGA https://www.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-43.html

Levenshtein distance is a metric [1] and discrete metrics can be indexed with BK trees, taking advantage of the triangle inequality. For a detailed explanation, see [2]. I know of no real world implementation of this, however. [1] https://en.wikipedia.org/wiki/Metric_(mathematics) [2] http://blog.notdot.net/2007/4/Damn-Cool-Algorithms-Part-1-BK-Trees

Here are a couple of articles about a homebrew FPGA (only one logic cell, but it's step in your direction). https://hackaday.com/2012/09/20/homebrew-fpgas/ http://blog.notdot.net/2012/10/Build-your-own-FPGA

Using the HyperLogLog algorithm to estimate the number of unique elements in huge datasets: http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinality-Estimat… Very cool algorithm used in (among other things) redis.

Firefox's Spidermonkey used to support a similar notation for JS objects. I can't remember when it was removed, but it was pretty obscure: http://blog.notdot.net/2006/9/Serializing-JavaScript-objects-with-circ… http://philogb.github.io/blog/2009/03/24/sharp-variables/ https://developer.mozilla.org/en-US/docs/Archive/Web/Sharp_variables_i…

Just for fun: http://blog.notdot.net/2012/10/Build-your-own-FPGA

Isn't this just log structured storage [1], essentially? [1]: http://blog.notdot.net/2009/12/Damn-Cool-Algorithms-Log-structured-sto…

This is called a quadtree. Here's a comparison: http://blog.notdot.net/2009/11/Damn-Cool-Algorithms-Spatial-indexing-w…

I was looking for the same thing, this article below has an incredible explanation of the algorithm using Python: http://blog.notdot.net/2009/11/Damn-Cool-Algorithms-Spatial-indexing-w…

http://blog.notdot.net/2010/07/Damn-Cool-Algorithms-Levenshtein-Automa… See the part about how you would intersect a trie and a levenshtein automata and find the ones that must be in both. It's not as hard as one would think (Tries are already DFA) You can also do it for other types of indexes.

If edit distance is bounded there appear to be algorithms that are linear in the input size: http://en.wikipedia.org/wiki/Edit_distance and http://blog.notdot.net/2010/07/Damn-Cool-Algorithms-Levenshtein-Automa…

I'm a little confused about what you're asking for, but this might help? http://blog.notdot.net/tag/damn-cool-algorithms

Nick Johnson did a Damn Cool Algorithms post [0] on this that inspired me to play around with HLL myself and do a little writeup [1]. [0]: http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinality-Estimat… [1]: https://github.com/sergeio/hyperloglog

so it looks like your doing a geohash for the geoindexing bure not using a hilbert curve[1]? Did you guys look into more flexible structures like an rtree or some of the other stuff from libspatialindex [2] which might be a bit more flexible (e.g. for finding what polygon a point is in) [1]http://blog.notdot.net/2009/11/Damn-Cool-Algorithms-Spatial-indexing-w… [2]http://libspatialindex.github.io/

Yes, you're right. We're not using SQL queries at the moment as that would be very inefficient, it was just as an example for a small dataset. I'm currently researching MVP's and reading on VP-trees, BK-trees [1], GNAT [2] and HEngine [3]. Do you have any advice? [1] http://blog.notdot.net/2007/4/Damn-Cool-Algorithms-Part-1-BK-Trees [2] http://www.vldb.org/conf/1995/P574.PDF [3] https://www.cse.msu.edu/~alexliu/publications/HammingQuery/HammingQuer…

So I think this is roughly what antirez is talking about with 'registers' for a counter. From the article: The standard error of HyperLogLog is 1.04/sqrt(m), where “m” is the number of registers used. Redis uses 16384 registers, so the standard error is 0.81%. My understanding is that the error is very small because you're averaging over 16,000 registers. Edit: also, I think you want the mean because it'll give you back a precise/fractional power of 2 that is pretty close to the actual count. The median would give you the nearest integer power of two, which is probably not a good estimate of the real count. Double edit: looks like there's some research around improving averaging by throwing away outliers: http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinality-Estimat…