33 Bits of Entropy

Hey all, I'm one of the authors of the Netflix and social network de-anonymization papers referenced, and the author of http://33bits.org. I also had the pleasure of having many talks with law professor Paul Ohm over the technical aspects of his excellent paper (which another commenter has posted a link to.) Paul's paper is a great example of people in the law community "getting it." (He happens to have a CS degree and is a hobbyist perl hacker!) We need more people like Paul, and we equally also need tech people who understand law and policy. I encourage everyone to give Paul's paper a quick read, gain awareness of the issues such why the current privacy laws are wrongheaded and what needs to be changed, and be on the look-out for ways to change things (as consumers, as tech influencers, as citizens). Cheers.

I am reminded of that "33 bits" blog that pops up here periodically. Every living person is uniquely addressable with 33 bits of the right information. "California" is worth 8 bits by itself. Male gets you another 1. Age within one year is probably 6 or so... Ah, here's that blog: http://33bits.org/ Its fundamentally changed the way I think about anonymity. (I used to think it might be possible.)

And one of my favorite old blogs! https://33bits.org/

It appears that you do not understand the basics of how this works: even if the print changes in an absolute sense what does not change is that many bits from one print to another are constant. Those are the bits that matter, and if there are enough of them then they can be used to track you across sessions, even across session using Tor and sessions not using Tor. Tie that all together and it may very well be possible to tie an upload using the Tor network to a particular user visiting some random website at a later date. You're leaking bits all the time and not all that many of them are required to uniquely identify you. See https://33bits.org/ for an easy to consume introduction.

Arvind Narayanan has made this observation in terms of the difficulty or impossibilities of anonymizing databases: http://33bits.org/ However, he didn't propose a particular list of 33 questions whose answers are maximally statistically independent. Even with things that are totally uncorrelated, you do get some chance overlap where people happen to have the same answers on a small number of them. For example, if we took the first 33 bits of the SHA256 of one's HN username, you and I happen to agree in positions 7, 8, 11, 13, 15, 17, 18, 19, 21, 26, 27, 28, 29, 31, and 32. (So we could say that each additional bit of that SHA256 value doesn't actually provide a full additional bit of distinguishing power relative to the previous ones.)

If you think that "pretty anonymous" is a thing, you should probably check out http://33bits.org/

If someone successfully deanonymizes the data, would that make AT&T liable for damages? You really don't need that many bits of entropy to accurately identify someone. relevant: http://33bits.org/

For all you PGP fanatics, consider (from http://33bits.org/2011/03/09/link-prediction-by-de-anonymization-how-w… a blog which I highly recommend) the sentence "During (...) 2007-2009, Shmatikov and I (...) showed how to take two graphs representing social networks and map the nodes to each other based on the graph structure alone—no usernames, no nothing." Tinfoil won't save you this time.

Read the citations. Some of the results are self contained, though. (eg the one from http://33bits.org/ )

"Even freeing up anonymized patient data seems to be met with opposition." Unfortunately, statistically, "anonymous patient data" is an oxymoron. Any useful amount of patient data contain enough information to deanonymize it to a great extent, and in conjunction with other data often fully deanonymize. http://33bits.org/2010/06/21/myths-and-fallacies-of-personally-identif…

Arvind Narayanan (who is also on HN as randomwalker) has published papers and given many talks about this fundamental concept: "the level of anonymity that consumers expect—and companies claim to provide—in published or outsourced databases is fundamentally unrealizable." It sounds like this is an example of that principle. It's not feasible for the public database to be both useful and preserve complete anonymity of the doctors. You can check out much more from his two websites: http://randomwalker.info/ and http://33bits.org/

It's actually pretty debatable whether this year's KDD Cup will really help the science of music recommendation: http://musicmachinery.com/2011/02/22/is-the-kdd-cup-really-music-recom… Because it's entirely anonymised, not just the users but the artists too -- c.f. Netflix's problems with deanonymization: http://33bits.org/2010/03/15/open-letter-to-netflix/ This means you can't use any interesting characteristics of the music itself, or the associated metadata, to aid the recommendations. All the interesting domain knowledge is stripped out, which likely means the best solutions still won't work as well as algorithms that use metadata (like Last.fm's) or content analysis (like Pandora's) or both, and certainly won't lead to any particularly interesting insights about what drives people's tastes. Disclaimer: I work at Last.fm

This wouldn't work anyways. Especially for datamining-is-our-core-competency company like Google it should be relatively trivial to match your old and new identity. There are plenty of ways how to do it. There are only 7 billion people, you just need 33 bits to uniquely identify someone [1]. On social networks you leave much bigger trail of clues (photos, timelines, locations, friends, activity patterns, likes/dislikes, writing samples, etc). Put together enough of vague data and your identity will pop out. Remember EFF's Panopticlick [2]? [1] http://33bits.org/ [2] https://panopticlick.eff.org/

IIRC, Larry Detweiler's sock-puppeting attempts on the cypherpunks mailing list were unmasked by other cypherpunks with this technique in the early 1990s. Detweiler, natively paranoid and geographically isolated in Colorado, had come to the conclusion that most of the prominent cypherpunks posters were sock-puppets of a single person he called "Medusa" — he invented the word "pseudospoofing" to describe the phenomenon, although "sock-puppet" is the more common current term — and resolved to use the same technique in response. I never read the actual word-frequency analyses, just Detweiler complaining about having had it used against him, which I assume he didn't imagine. There was a recent paper (two or three years ago?) that claimed that people could usually defeat this kind of linguistic analysis simply by trying to change their writing style. I'm skeptical that this is applicable in general. http://33bits.org/ is a pretty good blog on the unreasonable difficulty of anonymity in a computerized world.

NoScript, RequestPolicy, Adblock Plus, RefControl, UAControl, Do-not-track cookie-disabled Mozilla Firefox user here. Regulating Internet technologies to this level of pedantic granularity will ensure that spammers, scammers, crackers and fraudsters have an effective monopoly on privacy-busting technology. The incentive for software to implement correct technological solutions will be taken away if an honesty box Do-not-track approach is considered adequate. I want my browser to have a maximal number of reasons (including a multi billion dollar advertising industry) to address the technical concerns highlighted by Panopticlick[1] or privacy experts that "get it"[2]. These technical problems will be exploited by unwanted parties on the Internet. Exploitation will occur legally in other jurisdictions out of reach of EU laws. The full text of the directive can be found at §66 on page 20 of Directive 2009/136/EC at [3]. [1] https://panopticlick.eff.org/ [2] http://33bits.org/ [3] http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:337:00…

for any purpose, at any time, provided that following collection of such location and speed information identifiable to your Vehicle They store the data tied to your identity. A data breach (quite common these days...) would be a Big Deal. GPS tracks of everywhere you've gone in your car, ever? That's worth quite a bit of money in the right hands. He gave no evidence that they were not anonymizing the data properly, he just assumed they were not. Zipcode, birthday, gender: identifies 87% of Americans[1]. Your (Home,Work) gps tuple? Unique[2][3]. His assumption is quite safe; every "anonymized" dataset that's been released into the public (that I know of) has been de-anonymized. Why would this one be special? 1) http://arstechnica.com/tech-policy/news/2009/09/your-secrets-live-onli… 2) http://crypto.stanford.edu/~pgolle/papers/commute.pdf 3) http://33bits.org/2009/05/13/your-morning-commute-is-unique-on-the-ano… EDIT: In response to parent edit and below comments I have no proof of these, but factoids I believe to be true (so feel free to base a research paper on them :D) 1) To identify commuters: (Highway-Entrance-Location, Average-Highway-Entrance-Time, Highway-Exit-Location, Average-Highway-Exit-Time) -> some derived values: approximate (home,work), average speed, average driving aggression 2) Really, now that I think about it, any dataset where multiple gps tracks (for a single person) are tied…

For all practical purposes, so does any other way of producing a document from plain old desktop printing (http://33bits.org/2011/10/11/everything-has-a-fingerprint-%E2%80%94-do… to handwritten letters.

I highly doubt that it's possible to publish such data anonymously. There are less than ten billion people, about 33 bits are enough to identify everyone individually. If you publish a couple dozen genetic markers you almost certainly have very few people who share the same pattern. See http://33bits.org/2009/12/02/the-entropy-of-a-dna-profile/

Reidentification algorithms could still be used to identify students; see http://33bits.org/tag/de-anonymization/

You only need 32.6 bits of information to uniquely identify everyone: http://33bits.org/about/

On a related note, this has a very interesting significance in the world of privacy and anonymous tracking. http://33bits.org/about/

Also this: http://33bits.org/2009/05/13/your-morning-commute-is-unique-on-the-ano…

Please don't; the Netflix challenge was a a historically embarrassing privacy catastrophe. http://33bits.org/2010/03/15/open-letter-to-netflix/

I have to wonder what you'd see If you used style as ID. Styles change from time to time But style stays from line to line. Names can change and faces too But writing tells you who is who.[0] Many say they are a crowd But fewer do once lost their shroud. Traps and snares one will find Many more if kept their wits about their mind Still plenty that you see Hide their face behind IP. Just a thought. [0]: http://33bits.org/2012/02/20/is-writing-style-sufficient-to-deanonymiz… [1]: EDIT: It seems to me that it is very possible that not all accounts here, though not necessarily in this thread, correspond to a single individual.

> Clearly you have not spent enough time on 4chan. What's fun is that, thanks to the fact that 4chan happily embeds the Analytics/Recaptcha combo, Google may already know your creepiest fetish (along with your real name) in an instant. They can know what you say in 4chan by matching timestamps. Unless, of course, you are a so called "nerd" and you stay behind 7 proxies and have a clean, incognito browser. dredmorbius' mention of http://33bits.org/about/ definitely applies here. It's scary that having a name attached to other signals normally found in public profiles (schools, desires) will instantly yield an address for anyone determined enough. It's scary that it became so ubiquitous. Sometimes, I really miss the old internet.

http://33bits.org/about/ 33 bits (actually, 32.6) is enough to identify anyone personally. As others have noted, there are multiple cases in which people have been identified through their online postings, even when not officially identified, by various disclosures. It's among the reasons why those who take identity seriously are very, very careful about what they disclose about themselves online. The Google, YouTube, and G+ names issues is really, really, really bad. It's quite specifically what turned me from a very positive booster of Google to a pretty deep critic. And yes, while I make some use of Google's services, under various identities, it's an exposure I'm very consciously attempting to reduce.

Neat blog, with a catchy tag ("The End of Anonymous Data and what to do about it"). The previous article mentioned upfront also looks interesting - Everything Has a Fingerprint: The Case of Blank Paper (http://33bits.org/2011/09/13/everything-has-a-fingerprint-the-case-of-… especially the part about using run-of-the-mill scanners instead of SEM.

There's been some fascinating developments in this area in just the past few days. MySpace is now releasing similar data commercially through Infochimps: http://www.readwriteweb.com/archives/myspace_bulk_data.php NetFlix get slammed by the FTC: http://blog.netflix.com/2010/03/this-is-neil-hunt-chief-product-office… and Arvind & co summarize the latest research on why anonymization is (mostly) broken as a security measure: http://33bits.org/2010/03/15/open-letter-to-netflix/ Big Data is getting bigger...

Consider the Netflix and AOL search data, where people could (maybe) be identfied from "anonymized" data. Or the facebook gaydar thing. http://33bits.org/2008/09/30/law-review-article-about-netflix-paper/ harvesting emails, that is an established business model called...