Coda Hale

“Work is work” [0]. As an organization scales, an increasing fraction of employee time is spent on communication instead of work, leading at best to diminishing returns on headcount and sometimes to total gridlock. To the extent that an organization wants to derive marginal productivity from marginal headcount, it needs to find ways to get employees to talk to each other less. [0] https://codahale.com/work-is-work/

It sounds reasonable, but I think a key requirement of operating like this would be to have solid leaders that will guide the organization in the correct way. This is similar to the best blog post I've ever read on organizations by coda hale [1]. His perspective of viewing an organization like a distributed computing system is enlightening and highly pragmatic. Scaling organizations is hard. To scale well, you need to avoid contention on shared resources and continually work on force multiplier type projects. [1] https://codahale.com/work-is-work/

Read the article[1] linked in yock's comment for the gory details, but to sum up: instead of having to calculate the entire set of possibilities for a given hash, you only have to iterate just enough to get the first byte. Then you can do the same thing again for the next byte, and so on. You can't just fix this with some random timing or additional noise.. it's a very difficult problem to solve. It's telling that even the very smart person who did such a great write-up made a few mistakes again in initial attempts at demonstrating a corrected algorithm that actually blew the whole thing up again. Because it's so subtle, the odds are very good that most current systems are fallible to this. 1. https://codahale.com/a-lesson-in-timing-attacks/

Went through the SQL injection demo, and it recommends parametrized queries. Excellent. EDIT: Joined with Github, went through the password handling section, then saw this: http://i.imgur.com/H4h5FUY.png No no no no NO! Do NOT use SHA256 for passwords. https://paragonie.com/blog/2016/02/how-safely-store-password-in-2016 https://codahale.com/how-to-safely-store-a-password/ PBKDF2-SHA256 with 100k or more iterations? Okay, fine. SHA256 the cryptographic hash function not designed for password storage? Bad advice.

The points made here are covered in Coda Hale's post: http://codahale.com/how-to-safely-store-a-password/ While it recommends using bcrypt, if you augment that with "or scrypt or PBKDF2" the same principles apply.

Some caveats about Kafka replication are discussed here: https://aphyr.com/posts/293-call-me-maybe-kafka. It is important to include these in the "exactly-once" discussion... (hard to be exactly-once if messages are lost broker-side). Also, great discussion from Coda Hale about "CA" systems here: http://codahale.com/you-cant-sacrifice-partition-tolerance/

Maybe out of scope of the article, but for the section on rate-limiting he could maybe mention tptacek (and others') advice and recommend using bcrypt. http://codahale.com/how-to-safely-store-a-password/

> Set no maximum password length (250+ characters) It's important to note that if you take the classical advice to "use bcrypt" (http://codahale.com/how-to-safely-store-a-password/), your password will be effectively truncated at 72 characters.

Never understood the hate towards Java on HN. Java is not perfect by any means but every one of the other languages he mentions in that article (Perl, Python, Ruby, JavaScript) also has their shortcomings. People focus way too much on technology and not enough on the results. It reminds me of a great post by Coda Hale a couple of years back where he was explaining why he switched back to Java from Scala (which was hot shit on HN at the time): http://codahale.com/the-rest-of-the-story/ "The world has yet to take me aside and ask me for my opinion of it, and in the past few years I’ve found that it’s far more profitable to build things rather than tilt at windmills."

This is Bruce Schneier's favorite explanation of the blockchain: http://www.michaelnielsen.org/ddi/how-the-bitcoin-protocol-actually-wo… Comparing it to a "traditional DHT" is like kind of like comparing SQL and NoSQL. This is a good start on why that doesn't work: http://codahale.com/you-cant-sacrifice-partition-tolerance/

Great doc reference, thanks. I've always been impressed by RethinkDB and so was surprised to see the blurb above about being "CA". Good to see that's not actually the party line. Obligatory Coda Hale reference: http://codahale.com/you-cant-sacrifice-partition-tolerance/

The second one is a timing issue. You need to have an equality method that takes an equal amount of time on success or failure. Heres a good read on timing attacks in general: http://codahale.com/a-lesson-in-timing-attacks/

Worth a re-read: http://codahale.com/you-cant-sacrifice-partition-tolerance/

I mean this in the most polite and professional manner possible, please take five minutes and read http://codahale.com/how-to-safely-store-a-password/ . It will explain why "salts are useless for preventing dictionary attacks or brute force attacks." The entire article is excellent - and every colleague who I've ever pointed at it, has come away nodding their head and seeing the light. The key-takeway (but please, read the entire article) is: "It doesn’t affect how fast an attacker can try a candidate password, given the hash and the salt from your database." Salts only help you from precomputed dictionary attacks ("Rainbow Tables") - but, if someone is brute forcing you, the value of a salt just disappeared.

The solution is to use a better method of storing passwords. Hashes like SHA1 are designed to be really fast (great for hashing data but also great if you want to brute force). I think this is a pretty good overview: http://codahale.com/how-to-safely-store-a-password/

Bcrypt does not require your understanding. The most important thing is that you use a strong password hashing method -- of which bcrypt is the best-known, and an excellent choice. For a basic level of understanding, here's a slightly exasperated blog post that a lot of people link to: http://codahale.com/how-to-safely-store-a-password/

Great. I think `bcrypt` is another way of hashing passwords. Read this: How to safely store a password http://codahale.com/how-to-safely-store-a-password/

This is generally considered the best summary of the topic and bcrypt is strongly encouraged, to say the least: http://codahale.com/how-to-safely-store-a-password/ I don't think the concern is so much hacking your login than it is what happens if someone grabs your user table. That said, I suspect what you are proposing is stronger than average even among the HN crowd.

Straight link to the slides: http://codahale.com/codeconf-2011-04-09-metrics-metrics-everywhere.pdf This is a _great_ talk -- if you have any interest in writing code that other people will use, these are the kinds of ideas that you need to keep in mind.

It is amazing that a "senior" guy would resist something as simple as hashing a password. That said, salting a simple hash is not nearly as effective as people think. Given the computing power of modern GPUs, you should use bcrypt: http://codahale.com/how-to-safely-store-a-password/

To store a password securely, you use a hashing algorithm that isn't designed to be fast, i.e. bcrypt. Please read: http://codahale.com/how-to-safely-store-a-password/. Giving advice and in general debating about a specialized topic like crypto is a touchy business. With specialized topics in general, those without deep domain knowledge typically get things wrong. And in a topic like crypto, wrong advice can have significant negative consequences for those that follow it. The reason why Thomas is taking you to task on this one is that your original post talks about secret salts. That's a big red flag that there is a lack of understanding of underlying, fundamental security principles.

If a slave can continue serving reads whilst partitioned from a master that continues to accept writes then you cannot guarantee consistency. If a slave cannot serve reads when partitioned then you aren't available. If a master cannot accept writes when partitioned then you aren't available. See this excellent post from Coda Hale on why it is meaningless to claim a system is partition tolerant http://codahale.com/you-cant-sacrifice-partition-tolerance/. One love. - Lil' B

It has a lot to do with the weakness of MD5. You can compute the md5 has of every 6 character lowercase string in seconds on a modern laptop. There are also vast rainbow tables available. However mozilla have disabled all the accounts that used MD5 anyway. Unfortunately this does not prevent the real risk, which is to people who reuse their password elsewhere. Still, other hash functions such as the SHA512 they are currently using are also wildly inappropriate, as explained in the following excellent post: http://codahale.com/how-to-safely-store-a-password/

It's a huge field, basically distributed computing, burdened here with the glorious purpose of durable data storage. Any introductory text long enough becomes essentially a university-level computer science course. RADOS is the underlying storage protocol used by Ceph (https://ceph.com/). Ceph is a distributed POSIX-compliant (very few exceptions) filesystem project that along the way implemented simpler things such as block devices for virtual machines and S3-compatible object storage. Clients send read/write/arbitrary-operation commands to OSDs (the storage servers), which deal internally with consistency, replication, recovery from data loss, and so on. Replication is usually leader and two followers. A write is only acknowledged after the OSD can guarantee that all later reads -- including ones sent to replicas -- will see the write. You can implement a filesystem or network block device on top of that, run a database on it, and not suffer data loss. But every write needs to be communicated to replicas, replica crashes need to be resolved quickly to be able to continue accepting writes (to maintain the strong consistency requirement), and so on. On the other end of the spectrum, we have Cassandra. Cassandra is roughly a key-value store where the value consists of named cells, think SQL table columns. Concurrent writes to the same cell are resolved by Last Write Wins (LWW) (by timestamp, ties resolved by comparing values). Writes going to different servers act as…

Coda Hale's "work is work" is my favorite analysis of this topic, because of its focus on axiomatic mathematical upper bounds on productivity and how you can avoid hitting them: https://codahale.com//work-is-work/ The solution, as mentioned by other comments already, is for leaders to ruthlessly focus on keeping work efforts as independent as possible: > When presented with a set of problems which grow superlinearly intractable as N increases, our best bet is to keep N small. If the organization’s intent is to increase value delivery by hiring more people, work efforts must be as independent as possible. Leaders should develop practices and processes to ensure that the work efforts which their strategies consider parallel are actually parallel. Shared resources should be continuously managed for contention, and where possible, the resources a group needs should be colocated with that group (e.g., if the work involves a lot of design, staff a designer to that group). Combined arms doctrine isn’t just for soldiers.

I'm a developer, not a cryptographer, and like most crypto articles, most of this post is gibberish to me. I really tried to follow it in the beginning, but then I gave up, scrolled to the "what should you use instead", and then realized I didn't understand that either. If this is article is "easy" crypto, and I shouldn't roll my own crypto, then what should I do? "Just use elliptic curve crypto but make sure you choose the right curve! Here's an abbreviation for you, libsodium has it so you're all set" doesn't comfort me much. I don't have a good intuition of what a curve even is and why my sensitive data can't just go straight ahead like all the other data. I mean, when I call into libsodium with parameters I do not understand, then aren't I, for all intents and purposes, rolling my own crypto anyway? How is any of this of help? If I want to allow my customers to send me encrypted data via an untrusted channel, then what do I do? I need public key crypto for that, right? Basically, I want the public key cryptography version of Coda Hale's classic blog post about password hashing [0], does that exist? EDIT: thanks for all the comments folks, great insights! I particularly found [1] by CiPHPerCoder to be a great starting point (akin to Coda's password article) for all kinds of scenarios. I feel much more confident about this stuff now than I did 2 hours ago. [0] https://codahale.com/how-to-safely-store-a-password/ [1]…

This might be worth a read: http://codahale.com/the-rest-of-the-story/ http://codahale.com/downloads/email-to-donald.txt Having worked with Scala; I'd say I love it. Having worked with Scala; I'd say don't use it for most projects. It is not the best language to start learning FP. Wrap your head around a dedicated FP language and if you still feel the need for a hybrid FP/OO language Scala is definitely worth a look. However as it is a hybrid I've felt it made my code just too complex. Compared to Java there are a dozen more ways in Scala to fix something; and often you struggle to find the best way. This takes a lot of learning and it just isn't worth it. It's a great and elegant language in many ways, but I've come to dislike a lot of features of it. Simplicity is one of those things that make a beautiful codebase. See also (recently featured on HN); http://kotaku.com/5975610/the-exceptional-beauty-of-doom-3s-source-cod…

Integrated libraries cause one of .NET's biggest problems, though: the "if it doesn't come from Microsoft, it doesn't matter" effect. The Java ecosystem's openness provides you a lot of choice and has created a lot of innovation; Coda Hale had a good article on the topic[1]. I don't feel like there's a ton of non-Microsoft innovation in the .NET community--there's some iteration and a lot of (often poor--hello, NuGet) porting and adaptation, but there aren't many people doing something like Scalaz Streams on .NET. Even setting aside advancement and research, it's hard to really create a real competitors to Microsoft's existing solutions even when they're terrible. NHibernate is the "best" one I can think of and it's kind of trainwreckish. Outside of that--like, NancyFx has been around a long time at this point but hasn't really made a dent, whereas the JVM application server is rapidly losing ground--and it literally cannot come soon enough for my liking--to frameworks like Dropwizard or Play. You can fight the Man on the JVM and win; I'm not sure you can do so on .NET. That isn't to say .NET doesn't have some significant pluses. If you stay in the sandbox and can stomach having to deal with non-Unix command line tools or Cygwin (haaaa ha ha ha), it is really really nice. But the second you go off-roading, bring a helmet. [1] - http://codahale.com/when-formality-works/

I think, if you take the view espoused in http://codahale.com/you-cant-sacrifice-partition-tolerance/ , that P can be explained as the "the system is predictable across a partition." Geography and whether it's a "failover solution" doesn't change that the system is distributed - anything with more than one node is fundamentally distributed, and I'd argue that even two processes communicating on one instance can be considered as a distributed system as well. The system can choose which kind of predictable it wants to be: whether it sacrifices consistency in favor of being available (accepting reads and writes without knowing that reads are fresh or writes linearizable) or availability in favor of consistency (rejecting reads and writes that can't be confirmed). In the case of "master<=>master replication" as it's configured in most relational databases, I think the system tries to be AP: conflicted writes or stale reads are possible because barring special options like 2-phase commit, the replicas generally lag each other (as anyone who's tried to reconcile a MySQL split-brain situation knows, this can be a tremendous pain).

He's correct in that if you've selected bcrypt for key derivation, there's a good chance you could be doing things better (for one, its output is only 184 bits long; insufficient for AES256) where PBKDF2 works in a way where you can customize the output length. However, the point of the bcrypt argument is not that bcrypt is the best algorithm for certain things, but that it's (at a minimum) about four orders of magnitude better than most people's "secure" password storage algorithm: sha1(password). Because it requires both a salt and a work factor, even a dictionary attack is wildly impractical unless there's a massive flaw discovered in the algorithm. If developers are going to be trained to pick a specific algorithm for password storage, I'd much prefer bcrypt (no known flaws, many benefits) over sha1 or md5 (designed to be fast for checksumming, salt not required). Might PBKDF2 be a better choice still? Very possibly; I haven't done enough research to intelligently answer - and since this is crypto, I will not best-guess it. My real point here? The article attacks bcrypt as a key derivation algorithm, but I've never seen someone suggest it to be used in such an application. Even the post that started what you may call the bcrypt movement (http://codahale.com/how-to-safely-store-a-password/) is linked in the article, and it's titled "How to safely store a password". It is NOT titled "How to safely derive encryption keys". So yeah, I'm calling linkbait.