Dropbox Tech Blog

I've tried to sign up but you wouldn't take my password ("the input must contain at least one number or symbol"). Although a minimum length make sense, requiring a symbol doesn't. Would be great if you could remove it! See this for discussions on the subject: http://tech.dropbox.com/?p=165 http://xkcd.com/936/

Dropbox's approach here in the early 2010s (not sure if still used) was actually quite clever. Both official clients and third parties could open up a long-timeout HTTP connection that wouldn't be responded to until the web server was delivered a message in their internal message queue. Avoided the overhead of polling and allowed for extremely low latency, while letting clients still use their favorite HTTP library. https://dropbox.tech/developers/low-latency-notification-of-dropbox-fi…

Incorrect. Semantically, Rust always inserts bounds-checks, regardless of optimization level. The compiler _may_ then remove them as part of an optimization pass, iff it can prove the check is redundant. Do bounds checks even matter, even for very tight loops? This article [1] seems to suggest that removing the checks can give up to 10-15% speed increase under some circumstances. Worth it? I'd say no. [1] https://dropbox.tech/infrastructure/lossless-compression-with-brotli

Author here. We recently published a post about our last 4 years on SMR usage here: https://dropbox.tech/infrastructure/four-years-of-smr-storage-what-we-… Note that SMR technology just 5 years ago was rather nascent and a lot of software support was not great. Using SMR for us is possible without penalty only because of our sequential write workloads. We use a custom disk format along with libzbc, but libzbd now provides many advantages, which we are looking to adopt. I did want the QCon talk to have some super straight-to-the-point conclusions and these, I believe, have saved us the most since I have been on the team. Largely due to the sheer scale of managing such a system.

Not only that, but you probably need follow the sun if you want <30 minute response time. Given a system that collects minute-based metrics, it generally takes around 5-10 minutes to generate an alert. Another 5-10 minutes for the person to get to their computer unless it's already in their hand (what if you get unlucky and on-call was taking a shower or using the toilet?). After that, another 5-10 minutes to see what's going on with the system. After all that, it usually takes some more minutes to actually fix the problem. Dropbox has a nice article on all the changes they made to streamline incidence response https://dropbox.tech/infrastructure/lessons-learned-in-incident-manage…

I imagine following the tech blogs of certain companies - you can read about dropbox's engine rewrite here - https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-en…

This sounds great in principle and I saw this tried for several years for Dropbox mobile. It was a failed experiment though: > By writing code in a non-standard fashion, we took on overhead that we would have not had to worry about had we stayed with the widely used platform defaults. This overhead ended up being more expensive than just writing the code twice. https://dropbox.tech/mobile/the-not-so-hidden-cost-of-sharing-code-bet…

Nice work on the migration! I worked on the Dropbox coffeescript-to-typescript migration, and any sort of migration of that magnitude is non trivial :) We started out with the same approach as you guys, by introducing Typescript to the company, making it possible to write code in Typescript, and encouraging teams to use it. We had an OKR to convert all of our Coffeescript to Typescript by the end of the year, though, so eventually we had to go with an automated approach to convert the rest of the files, which you can read about here: https://dropbox.tech/frontend/the-great-coffeescript-to-typescript-mig…

Dropbox still has the best syncing tech [1][2] which handles complex situations with lots of files and responds instantly. That alone is worth it if you need it. However the rest of the product is crap. Dropbox had a solid business with a "magic folder that syncs" but has been chasing new features and product lines with little success. Even the enterprise offering was far behind Box.com for years and now has tons of competition. 1. https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-en… 2. https://dropbox.tech/infrastructure

Dropbox has a fairly exhaustive post about moving away from Coffeescript that involves decaffeinate: https://dropbox.tech/frontend/the-great-coffeescript-to-typescript-mig… I do hope for their own sake that Bubble looks at migrating away. At scale an untyped de facto abandoned 'language' is going to catch up to them sooner or later and limit their ability to iterate safely at scale. [Disclaimer: used to work at Dropbox, but had no involvement in this project other than massive appreciation for the amount of effort involved]

gRPC is one of the best decisions we've made at our company. Here's the longer blog post - https://dropbox.tech/infrastructure/courier-dropbox-migration-to-grpc, but some things: 1. Performance 2. It's hard to make changes that are backwards incompatible via protobuf (reduces significant source of bugs) 3. Great, standardized observability for every service. Small services don't really need too many custom metrics, since we log a LOT of metrics at the RPC layer 4. Standardization at the RPC layer lets us build useful generic infrastructure - like a load testing framework (where users only need to specify the RPC service, method, and parameters like concurrency, RPS).

Great question! We developed and deployed Lepton to losslessly encode JPEG image files. Lepton continues to deliver substantial storage and cost savings every year. You can read more about it here https://dropbox.tech/infrastructure/lepton-image-compression-saving-22…

Another example would be Dropbox which quite recently published this great blog article about "Rewriting the heart of our sync engine" in Rust [1]. [1] https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-en…

I did not have a great experience with that approach. The now (in)famous DropBox article describes why way better than I ever could: https://blogs.dropbox.com/tech/2019/08/the-not-so-hidden-cost-of-shari… My biggest pain points were tooling. For all of Xcode’s faults, C++ and Objective-C debugging and intermingling wasn’t one of them. Android, though, was another story.

Do they have to? Sync can still be improved a lot, and I think whoever does it best will be able to take over most of the market. Dropbox's Infinite could potentially do that. https://blogs.dropbox.com/business/2016/04/announcing-project-infinite…

I had asked a similar question on Quora [1] few years back. Here is the summary: - Facebook: https://www.facebook.com/Engineering?sk=notes - Google: http://google-engtools.blogspot.com/ - Twitter: http://engineering.twitter.com/ - Linkedin: http://engineering.linkedin.com/blog - Dropbox: https://tech.dropbox.com/ - Instagram: http://instagram-engineering.tumblr.com/ - Netflix: http://techblog.netflix.com/ - Flickr: http://code.flickr.com - Etsy: http://codeascraft.etsy.com/ - Yelp: http://engineeringblog.yelp.com/ - Heroku: https://blog.heroku.com/ - Airbnb: http://nerds.airbnb.com/ - Bitly: http://word.bitly.com/ - Evernote: https://blog.evernote.com/tech/ - Highscalability: http://highscalability.com/ [1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-bes…

> However, I know that Dropbox is one of the biggest Go users and it isn't in this success stories list. Anyone knows more about it? Maybe the go devs see them as traitors for using rust for their core components. :P They were having memory consumption issues with go, so I'm not sure that would be the best thing to add to a list of success stories. [1]: https://blogs.dropbox.com/tech/2016/06/lossless-compression-with-brotl… [2]: https://news.ycombinator.com/item?id=11283685

Such a great timing for this post, as Dropbox announced that they are shutting down Mailbox: https://blogs.dropbox.com/dropbox/2015/12/saying-goodbye-to-carousel-a…

Dropbox also ruined one of my favorite mail client on Mac: Mailbox. Somehow they did an update from 0.4x to 0.7x which made the client orders of magnitude uglier, crash more often, have dozens of rendering issues which never happened in 0.4x. They did this without communicating anything in their blog[0], and meanwhile totally ignores all the complaints in Dropbox forum[1]. [0]: https://blogs.dropbox.com/mailbox/ [1]: https://www.dropboxforum.com/hc/en-us/community/topics/200211215-Mailb…

This is great. Also see: https://zulip.org And the blog on why Dropbox decided to OSS it: https://blogs.dropbox.com/tech/2015/09/open-sourcing-zulip-a-dropbox-h…

Well, for one, Dropbox is claiming that it was not a hack at all. See: http://techcrunch.com/2014/01/10/dropbox-offline-hacker-group-claims-c… and also see dropbox's blog post, explicitly stating that it "was caused during routine internal maintenance, and was not caused by external factors" https://tech.dropbox.com/

So you're going to INSERT directly from Javascript? Seems like potential security issues but back on point.. Check out dropbox.js blogpost: https://tech.dropbox.com/?p=345 Even better seems to be Firebase which touts itself as "Dropbox for your app's data": http://www.firebase.com/how-it-works.html

And the process for you to stop being such a trolling asshole ... well there is none. You're stuck being an asshole. You may not think this is an important point, but that doesn't mean that people who actually understand software development do. See, even those who use it can't actually make a real argument against this other than it was easy for them. Whatever that means ... https://tech.dropbox.com/?p=361

The Dropbox post on passwords (and their password strength estimation tool) is much better at describing what strong passwords actually are. http://tech.dropbox.com/?p=165 The advice from LinkedIn like "Don't use a word from the dictionary" and "Substitute numbers for letters that look similar (for example, substitute “0″ for “o” or “3″ for “E”." are security theater.

Model testing and swapping is one of the surprises people really appreciate DSPy for. You're right: prompts are overfit to models. You can't just change the provider or target and know that you're giving it a fair shake. But if you have eval data and have been using a prompt optimizer with DSPy, you can try models with the one-line change followed by rerunning the prompt optimizer. Dropbox just published a case study where they talk about this: > At the same time, this experiment reinforced another benefit of the approach: iteration speed. Although gemma-3-12b was ultimately too weak for our highest-quality production judge paths, DSPy allowed us to reach that conclusion quickly and with measurable evidence. Instead of prolonged debate or manual trial and error, we could test the model directly against our evaluation framework and make a confident decision. https://dropbox.tech/machine-learning/optimizing-dropbox-dash-relevanc…

Elsewhere in the comments someone linked to this: https://dropbox.tech/infrastructure/lossless-compression-with-brotli It looks like dropbox experimented with disabling bounds checks in their huffman coding impl, and found that using the unsafe pattern increased throughput from 224 MB/s to 249 MB/s (11%-ish faster.) We don’t even need to hypothesize about whether webp would have elminated bounds checking, we can see that other companies arrived at the same conclusion: Disabling it can be worth it if you’re quite sure you’ve gotten the up-front checking right. We can imagine that if Dropbox went to prod with the unchecked huffman implementation (never mind that that article isn’t about webp in particular), we could imagine they could easily have the same bug. And I don’t think a naive code review saying “unsafe is bad” would have stopped them from doing it: they clearly did the work to show why it’s worth it.

> They're always renaming and moving files and folders to keep things organized. Ah, there we go! It's their won fault for assuming a filesystem can be used for basic file and folder changes. /s On a serious note, it turns out that replicated filesystems as seen in OneDrive or DropBox are hard to engineer. Astonishingly difficult to get right. "Formal proof verifiers" level of difficult, otherwise customers will lose data in everyday circumstances. It's happened to me, it happened to all of my coworkers, and it has happened to your customers. Data loss with a poorly engineered replication system isn't some rare corner-case, it's the typical case, and it is a miracle if data isn't lost! A few years ago DropBox rewrite their sync engine and wrote a great blog article about all the complex scenarios they had to handle: https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-en… OneDrive has done none of this. NONE. I guarantee it. It loses data if you look at it wrong. If you call Microsoft Support they'll tell you not to look at it like that. I do all of my work on local C:\ paths and copy files to SharePoint or OneDrive only if forced to collaborate with colleagues. This rare interaction with OneDrive causes data loss about 5% of the time, often enough that I notice.

> So swapping languages, yeah maybe, but I expect of more practical use would be the situation where you inherit a legacy codebase in an ancient version of a language or framework that hasn't been loved in a long time. I saw this so many times when doing dev team for hire work. I'm exploring this problem space - this is a wide spread pain point across almost all companies that are older than 3 years old. I've seen this at tech startups as well as very large companies and anything in between. Dropbox is arguably a top tier engineering organization that probably manages tech debt as responsibly as can be expected and they still had to make major investments to move their codebase forward from various eras of web tech 1) https://dropbox.tech/frontend/the-great-coffeescript-to-typescript-mig… 2) https://dropbox.tech/frontend/edison-webserver-a-faster-more-powerful-… Everyone else is much worse off so the investment required to move forward is usually immense. This leads to full rewrites. Which is nice but error prone and sometimes entails huge opportunity costs > Obviously you'd want to do boat loads of testing and there may well be manual work left to do afterwards, but I think it would be the kind of manual work that felt like you were polishing something new and clean and beautiful rather than trying to apply bits of sticky tape to something unmaintainable. Agreed. In my opinion, now's a great time to get started with a semi automated approach…

> E.g. postcriptaluminumengagement vs kug45l2wx If you pick from a dictionary of 250,000 words, that is around 18 bits. So three randomly chosen words strung together give you around 54 bits. On the other hand, an alphanumeric character is around 6 bits, so 9 of them strung together is about 54 bits. Assuming your dictionary was 250,000 words, both of the passwords you posted were about the same quality (until you posted them - now they're again about the same quality, but much less). > Or are there also dictionaries that contain combos of words?! Your software just concatenates words and other well known sequences (123, zxcvbn). Check out the great password entropy checker called zxcvbn: https://github.com/dropbox/zxcvbn https://dropbox.tech/security/zxcvbn-realistic-password-strength-estim… https://www.bennish.net/password-strength-checker/ (Note: don't enter production passwords into random websites, needless to say...)

This was 5+ years ago so I'm a little hazy on the details. A lot of it was just plugging into our infrastructure for config, service discovery, rate limiting, logging, etc. - Logging the exact metrics we wanted to our metrics service. (We wrote a bunch of Lua code to do this.) - Loading the set of backend services from Zookeeper. (We had a sidecar process that would periodically sync from Zookeeper to a config file, then call `nginx reload`.) - Routing requests to different datacenters based on where the user's data was homed. (We had Go libraries to do this, and it would have been nice to be able to just call into that. I think we ended up putting this functionality in the sidecar.) - Changing the way we streamed a response based on an HTTP header from the application. (We ended up writing a C module for this.) - Changing the backend selection algorithm from "least connections" to "best of 3 random choices". Nginx added support for this in 2018, apparently. There were ton of things we solved with Lua, which wasn't ideal. Since that's the only Lua in our codebase, everyone who touches it has to make a mental switch to remember all the weird edge cases (as you do with any language), plus 1-based array indexes, plus no static types. Plus, any Lua or C you write has be contorted to fit Nginx's multi-stage request/response architecture. If you're writing a plugin that is meant to play nice with other plugins, then the contortions may be worth it. But for us, a proxy…