Bret Taylor

I'm always reluctant about going completely schemaless. Reminds me of the blog post by FriendFeed about how they use MySQL, highly recommended: http://bret.appspot.com/entry/how-friendfeed-uses-mysql I feel like like there needs to be a better middleground for having some schema but being able to augment it easily with metadata that you're not querying against (yet). Then later extracting the metadata into queryable columns. I wrote a post outlining some ideas of how to do this: https://github.com/shazow/everything/blob/master/idea/arbitrarily-stru… I've only implemented bits and pieces of this in practice, huge convenience so far.

I'm not sure why andreyf was being down-modded, the graph on the original blog post at http://bret.appspot.com/entry/tornado-web-server uses a subtly different shade of blue. This is a clear attention to details that is worthy of at least a small note in a community such as ours :-)

Nicely done ~ http://github.com/btbytes/teh/tree/master/webapp.py and shows just how versatile the framework is by the number of different libs being used. Bret still hasn't released his sample code ~ http://bret.appspot.com/entry/experimenting-google-app-engine#disqus_t… so this is good for users. The feed doesn't work. The best bit Bret showed was the decorator '@administrator'. My version uses webpy, sans disqus. Work on that some other time ~ http://seldomlogical.com

Perhaps you'd enjoy this following article, then: http://bret.appspot.com/entry/experimenting-google-app-engine ;-)

I'm a GAE user for a personal app -- basically I use it for serving comments on my personal blog, similar to whatever Disqus is doing but I prefer the control. I also built on it 2 other personal apps that I later shut down. IMHO, GAE is great for quick prototypes. Even for apps built for personal usage is great as the free quota is still reasonable, and compared to shitty PHP hosting services it's heaven-like. As far as the free quota goes it's also a lot better than Heroku. However, I feel very restricted on GAE. My current project at work that pays my bills would never fit into it. Even if it did, because of the stuff we do asynchronously and because of how Google charges for API calls, the bills would go through the roof. the only PaaS offering that can scale the data-tier Yes it is. However, you can also scale the data-tier by yourself. Yes, it takes engineering, but it really isn't that difficult if you're planning ahead a little, which you have to do with GAE anyway. I think this old article from 2009 about how Friendfeed used MySQL to do sharding is a classic by now, but in case you haven't read it: http://bret.appspot.com/entry/how-friendfeed-uses-mysql My biggest problem with GAE is that it is hard to migrate from when it no longer suits your needs. Not impossible, not extremely difficult, but difficult nonetheless taking away precious resources you'd rather invest in something else. But as far as PaaS goes if you're in to that, it works great.

We're (Inkling) successfully using Cassandra in production to store most of our data. I wrote the first version of our data store modeled off of FriendFeed's architecture (back when we were five people). There is an excellent blog post about it here: http://bret.appspot.com/entry/how-friendfeed-uses-mysql When Cassandra came along, we had long debates internally about the risk of hitching ourselves to their wagon. Eventually we decided that an open source project with a similar (enough) architecture to what we'd been building ourselves was preferable to maintaining a python library in-house. So far we have no regrets. We have had no production issues and we have an architecture that is built to handle a large amount of incoming data (every note and highlight you take in an Inkling book is synched to our servers). We had already stripped away most SQL functionality by using FF's architecture, so we didn't lose much. And we gained a lot - the ability to tune the durability of different writes depending on the use case, far better fault tolerance, a conceptually simpler indexing strategy, etc. It definitely has a learning curve - error and status reporting is not wonderful, you generally need to be over-provisioned as a strategy, etc. But I'm happy we learned those lessons sooner rather than later - I'd hate to scramble to implement Cassandra in a strained production environment (my sympathies to a certain website who suffered through this).

Yup. See http://bret.appspot.com/entry/how-friendfeed-uses-mysql for an example. MongoDB is more convenient to program because it's all built-in.

The irony of this is that it's actually quite easy to build a document store on top of PostgreSQL that performs very well indeed. I tried it. I used a similar approach to the one described at [1]. You get the advantage of years of experience (never underestimate this!), MVCC, transactions, consistency and replication as well. (yes I know it doesn't "perform" as well as the other NoSQL stores but performance is not without tradeoffs [2]) [1] http://bret.appspot.com/entry/how-friendfeed-uses-mysql [2] http://www.caradvice.com.au/wp-content/uploads/2006/12/FerrariCrash4.j…

"How FriendFeed uses MySQL to store schema-less data" http://bret.appspot.com/entry/how-friendfeed-uses-mysql

It's similar to how FriendFeed uses (used?) MySQL: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

Reminds me of this article on how friendfeed implemented schemaless storage on top of MySQL: http://bret.appspot.com/entry/how-friendfeed-uses-mysql I always thought it was a very clever design.

Friendfeed did something similar: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

Brett Taylor discusses this issue in his tech talk on Tornado (http://bret.appspot.com/entry/tornado-tech-talk). He uses a hybrid approach: Use synchronous programming when requests block for a short time (fetching data from local db or memcache) and async programming when something blocks for a long time (calling out to third party services, long polling requests). He does not go all async because programming with callbacks or deferreds is more complicated than straight line synchronous programming. Brett also mentions that the instances of the server are created once and reused. He does not fork an instance of the server per request.

This seems to be a pretty good implementation of FriendFeed's schema-less MySQL ( http://bret.appspot.com/entry/how-friendfeed-uses-mysql ) and this option would probably beat most of the simple key-value stores in terms of perfomance, backup and replication. One should note thought that more modern NoSQL databases aren't only key-value stores. For example, Redis offers KEYS command which can scan the store by a pattern. Tokyo Tyrant (another "key-value" store) offers Lua scripting.

"In general, it seems like Twisted is full of demo-quality stuff, but most of the protocols have tons of bugs." http://bret.appspot.com/entry/tornado-web-server#comment-16314810 That's not "wasn't for us" that's a very vague comment insulting the project. If that were true, I think it is fair to at least describe what they were -- or the nature of them -- or something.

Chat demo running live: http://chan.friendfeed.com:8888/ Bret's blog post: http://bret.appspot.com/entry/tornado-web-server

As I recalled, FriendFeed uses MySQL as the "datastore" backend, but with very different table layout. Check out this article: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

you could do an in between: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

Python not ruby, but the language on top is a minimal piece in their setup: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

http://bret.appspot.com/entry/how-friendfeed-uses-mysql This is a really good article that can get you started with this idea. You would have to develop your own solution but a lot of good ideas here.

Good talk. Interesting that Facebook uses MySQL mostly as a key-value store to represent directed graphs. There was a discussion recently about how FriendFeed does essentially the same thing. http://bret.appspot.com/entry/how-friendfeed-uses-mysql. I'd like to see a bit more about how they implemented this.

I'm not a 10x coder, but over the years I've learned to deal with buggy 3PLs by avoiding them, or encapsulating them once and for all. The problem with buggy 3rd-party libs is that the bugs are random. They quite literally keep you dumb by forcing to you memorize random trivia instead of new concepts. Rewriting the same functions yourself in a better and more efficient way teaches you a lot, especially about 1) how to spot good libraries, and 2) when not to write it yourself. If you look at the code of really effective hackers, the one thing they have in common is how small the code is compared to what it does. Look at web.py, or bret taylor's datastore built on top of mysql: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

FriendFeed has an in-house framework inspired by web.py and Appengine's webapp. It was written by Bret Taylor, who was the PM of appengine when it started, but is now working for FriendFeed. He mentioned that he might open source it, but hasn't done so yet. I encourage people to put some pressure on him to do this =)! http://bret.appspot.com/entry/experimenting-google-app-engine EDIT: I'm not sure who wrote the framework, so I was wrong to say that Bret wrote it. In any case, it was written by FriendFeed developers.

friendfeed was built in 08, before amazon had released EBS as a feature. there is no way in hell the instance-store disk i/o would have kept up with friendfeeds mysql demands (you may remember the post Bret himself wrote on their nifty approach to mysql: http://bret.appspot.com/entry/how-friendfeed-uses-mysql ) Frankly, even EBS probably wouldn't have. Plus, back in 08 the larger ec2 instances were still pretty low on ram compared to what you could cheaply cram in a colo'd server, putting even more pressure on the disk i/o. I think Bret's really wishing what almost everyone is, that these problems would just go away and you could pay a reasonable premium for that, but we're not there yet and certainly weren't in 2008. I don't think he really checked his numbers on this one. edit: however there is a happy medium. managed service providers will gladly charge you ec2 prices (250/month for a low end server, 500/month for a beefy one) while handling all midnight colo trips for you.

The OP's suggestions are pretty bad, as jasonkester points out. But for a specific class of problems (ones that are both performance or JOIN intensive and require the awesome indexing/sorting capabilities in SQL solutions), schema-less MySQL, when done "correctly" is awesome. I designed an application that needed to do graph traversal in real-time, so normalized SQL wasn't an option for me. I tried various NoSQL solutions, but they all either lacked important features or ran insanely slow. I ended up settling with a design similar to how FriendFeed uses MySQL: http://bret.appspot.com/entry/how-friendfeed-uses-mysql Not only was the solution faster, but it made code so much more concise, because you can just serialize Python objects to the database rather than write a huge database abstraction layer. For me this is even more valuable than the performance ramifications. If you're using Python+MySQL, be sure to check out Tornado's database wrapper (http://github.com/facebook/tornado/blob/master/tornado/database.py) because it handles all sorts of sharp corners that straight up MySQLdb won't (like working with binary data, reconnecting when necessary, etc.)