Neustar Research

HyperLogLog is a really interesting and powerful data type. Though a few things seem missed in the article. HyperLogLog has been around I believe over 10 years now. It's been supported for Postgres for over 5 years [1] now as an extension[2]. It is great to see it growing further with Facebook adding support for it for Presto. [1]. https://research.neustar.biz/2012/10/25/sketch-of-the-day-hyperloglog-… [2]. https://www.citusdata.com/blog/2017/06/30/efficient-rollup-with-hyperl…

Neustar Research has a good article on this which may be of interest to you. However, I think the approach they took was still based on the inclusion-exclusion principle (still a good read though). * https://research.neustar.biz/2012/12/17/hll-intersections-2/ Another approach if you have the flexibility of changing your implementation, is to use a different type of data-sketch that is more amenable to set expressions. There is some discussion (and references) here: * https://datasketches.github.io/docs/Theta/ThetaSketchFramework.html * https://datasketches.github.io/docs/Tuple/TupleOverview.html

>Are there any major ones I'm missing, or even better, any lesser known ones I should be aware of? MinHash. I also highly recommend this blog: https://research.neustar.biz/tag/sketching/

To anyone interested by differential privacy I would recommend this article: http://research.neustar.biz/2014/09/08/differential-privacy-the-basics… It does a very good job of explaining the basics. They also published a follow up with the quite popular taxi dataset: http://research.neustar.biz/2014/09/15/riding-with-the-stars-passenger…

I'm not able to review the data yet but I wonder if TLC took in any of the feedback to better anonymize the data. [0] Also, I would love to see an analysis of whether traffic is actually getting worse compared to that of last year. This claim was made by mayor de Blasio as a reason to cap Uber rides. [0] http://research.neustar.biz/2014/09/15/riding-with-the-stars-passenger…

@Rurban (Reini Urban): wrt to your first point, general purpose hash functions being modeled as PRNGs, I believe that statement to be correct. It is definitely the case when analysing hash function behaviour in contexts such as bloom filters, hash tables etc. With regards to the definition of perfect hash functions - if you had read the sentence before the one you have quoted, you would have notice that the formal definition for a perfect hash function has been given in the article as: > "In general there is a theoretical hash function known as the perfect hash function for any group of data. The perfect hash function by definition states that no collisions will occur meaning no repeating hash values will arise from different elements of the group." It then goes on to talk about the practicalities of perfect hash functions (deriving them etc), then makes a statement that generally a perfect hash function is one that has the least collisions - I agree that it is an intentional weakening of the formal definition, but it is generally the most practical one - as there is no guarantee that one can always find a "perfect hash function" for any arbitrary set of values in sub-polynomial time. ---- wrt the collection of hash function, take note that the article is broken into the following sections: 1. What is a general purpose hash function 2. Designing and implementing general purpose hash functions 3. Uses of hash functions 4. Example general purpose hash functions…

If you dig through to the source reddit posting [1], you can see that the post only really puts 6 different individual taxi IDs out there for visualization. To me, only the first 4 seem like a good visual fit for the prayer times, and the overall trip heatmap [2] suggests that they may just be part of a larger pattern of eating/taking a break at sunrise, noon, and sunset for all cab drivers. While the blog posting is begging the question of whether or not the data release contains personal data as a result of these findings, much more invasive findings have already been published (and posted to HN) with this NYC Taxi data, like corroborating individual trips by high profile people. [3] Notwithstanding the privacy issues and questionable methods used to obfuscate the data, I personally think that the release is a great step in the right direction for open data. [1] https://www.reddit.com/r/dataisbeautiful/comments/2t201h/identifying_m… [2] https://i.imgur.com/lyK0qTI.png [3] http://research.neustar.biz/2014/09/15/riding-with-the-stars-passenger… EDIT: For better or worse, deducing ethnicity, country of origin, and/or religion is probably much easier based on this data set [4]. People have come up with analyses like this [5]. The data analysis is great, but my fingers are crossed that tabloid newspapers and their ilk don't pick this up and run off xenophobic, fear-mongering articles. [4]…

Probabilistic Counting with Stochastic Averaging : https://research.neustar.biz/2013/04/02/sketch-of-the-day-probabilisti…

One of the ways we count distinct elements in an unknown amount of data--probabilistically! You may have heard of Bloom filters and the like, and this is a similar family of algorithms. https://research.neustar.biz/2012/07/09/sketch-of-the-day-k-minimum-va…

Another good explanation, this one with a javascript simulation of HLL: http://research.neustar.biz/2012/10/25/sketch-of-the-day-hyperloglog-c…

It is a bit unsettling that this information is out there, but I agree that it is fairly obvious that they have this data. And as long as they are not exposing the individuals, I don't see it as irresponsible to publish something like this. There was a related story published recently, NYC Taxicab Dataset Exposes Strip Club Johns and Celebrity Trips http://research.neustar.biz/2014/09/15/riding-with-the-stars-passenger…

Only if you can/do meaningfully anonymize the data. Eg, knowing the subnet of most/all Norwegian isps, it's trivial to recover ips that are simply hashed (probably even with salt), similarly Norwegian phone numbers are only eight digits, so any kind of deterministic mapping is likely to be too trivial to actually amount to anonymization. Also rember that one of the goals is to avoid illicit linking - so being able to verify that ip n.n.n.n is the same as slow_hash(salt+other-ip) won't fly as "not storing". In general, anonymizing data in sparse populations is tricky - where "small" can be quite large. Just imagine building a bitfield of variables like: sex,age +/-50;2 bits. Rough location (easily 6 bits), browser (2 bits), mobile? 1 bit - that's already 12 bits etc. See also NYC taxi dataset, eg (not the article I had in mind, but seems to cover similar points): https://research.neustar.biz/2014/09/15/riding-with-the-stars-passenge…

I am extremely glad that most comments here see the inherent craziness of this idea. I am a hardware designer who participated heavily in mining since Jan 2011, and gave up completely on Bitcoin in summer of 2014. The big problem that Balaji is completely ignoring is that SHA256 cores implemented in silicon have extraordinarily high power consumption relative to their size (mm2). This is due to the incredibly high signal switch rate inherent to implementing avalanche-style ciphers [1] in hardware. Perversely, most modern semiconductor processes optimize for static (leakage) power, since modern silicon designs are more vulnerable to this than to dynamic power. No rational semiconductor maker will want to include this IP on chip, because it will drastically complicate the power distribution metal layers. Additionally, it will require more expensive packaging, since you will need many more power and ground pins to source/sink the required current. I know this first hand after building a mining ASIC myself. I would venture a guess that most SoC chips would cost significantly more due to the extra engineering time and packaging cost needed to include this IP. This entire concept is a non-starter for me. [1] http://research.neustar.biz/2012/02/02/choosing-a-good-hash-function-p…

This article mixes up a couple different issues. First, internal controls: what can employees do when they access the user database? Beginning startups probably don't have a lot of formal policies and you're relying on good judgement of the founders and first employees. As a company grows, they need to guard user data with access restrictions, restrict root to a few trusted sysadmins, and put in place clear rules about what what employees with legitimate access to the database are allowed to do. (And of course if your database gets owned it's all for nothing, so you can't protect privacy without good security.) It sounds like Uber might be behind on this, relative to their scale. Second, what can you do with aggregate reports generated from user data? Facebook and OkCupid and Uber posted research articles to their blob summarizing some of their users' overall characteristics. When it's okay to do this sort of research is debatable, but it's a different issue from invading the privacy of individual users. Unfortunately taking a regular taxi may not help [1]. That was screwup rather than intentionally revealing user data, but by now everyone should know that publishing anonymized records is really hard and probably shouldn't be done at all. [1] http://research.neustar.biz/2014/09/15/riding-with-the-stars-passenger…

You can probably trust Uber just about as much as you can trust the TLC. The only real difference is that Uber has a bit more real--time data they can mine. Even if you only pay for your cabs in cash, your privacy can be compromised by pick up/drop off point meta data (http://research.neustar.biz/2014/09/15/riding-with-the-stars-passenger… Then what about all the other plethora of apps and OS features that do background location tracking. Can we trust them? I think this problem really goes beyond Uber and touches on something fundamental about how we can maintain privacy when everyday technology asks for so much more personal information to operate..... Maybe we need more robust data privacy laws. Maybe we need better technology that allows for identity obfuscation or anonymity. Maybe we need to mature as a society so transparency is less likely to lead to harassment, violence or discrimination. Maybe it is a combination of all of the above. I just don't think you should leave it up to Uber.