MyNoSQL

These are all fantastic advise for any open source project. A couple of days ago, I was arguing that for NoSQL projects in particular there is an additional required step: making sure that it is easy to be used from major frameworks and that this integration is "as seamless as possible" (you can read the whole thing here http://nosql.mypopescu.com/post/299775877/nosql-to-people). With all the NoSQL solutions around, right now it is quite difficult to pick the one that fits your problem. Making it simple to try it out will generate a lot of use cases and that will make further adoption even easier. :- alex ps: (excuse and/or ignore what probably can be seen as a plug) on MyNoSQL (http://nosql.mypopescu.com) I am trying to follow some of the most promising NoSQL projects and Redis is one of them. And according to the stats, there is a lot of cool activity around Redis.

Again, shoving everything into one file is mbox, which is a recognized file format (if not database), and it does not help things that much. I have wondered the Sqlite thing myself beyond my one criticism. If anything, you would love K9 Mail on Android, which does this out of necessity. And some people have made SQL and noSQL email stores. http://nosql.mypopescu.com/post/3083070885/couchdb-as-email-store https://www.npmjs.com/package/haraka-nosql http://www.dbmail.org/

Unknown is relative, but here are some of the rss feeds I'm subscribed too: http://nosql.mypopescu.com/ http://lambda-the-ultimate.org/ http://eli.thegreenplace.net/ http://www.stefankrause.net/wp/ http://mechanical-sympathy.blogspot.com/ http://www.cowtowncoder.com/blog/blog.html http://highlyscalable.wordpress.com/ http://hunch.net/ http://perspectives.mvdirona.com/ http://www.scottaaronson.com/blog/ http://www.softwarebyrob.com/ http://blog.mikemccandless.com/

update just found a great article on scaling with CouchDB using Lounge http://nosql.mypopescu.com/post/683838234/scaling-couchdb and a company that provides the service Cloudant: https://cloudant.com/ Here's another shard manager project, Pillow: http://knuthellan.com/2010/07/14/feature-prioritization-for-pillow-the… I think the design philosophy for couch is keeping smaller pieces of Big Data synchronized. The sum of all those small pieces is like one larger database.

Ryan King gave a good summary in his interview. (http://nosql.mypopescu.com/post/407159447/cassandra-twitter-an-intervi… Basically, Cassandra has the best answers to the important questions: How will we add new machines? Are their any single points of failure? Do the writes scale as well? How much administration will the system require? If its open source, is there a healthy community? (And I would add, is it _fast_ as well as scalable?)

While SQL can scale, I think this argument is a little spurious. I truly dislike the NoSQL stance of "never SQL;" it has its place, and its place isn't necessarily at the twitters or the facebooks of today. SQL scales very well with datasets that make sense for RDBMS. CRUD style applications. Core business apps. Data that doesn't necessarily need to be mined furiously. Trying to shoehorn, say, a high-volume message system or intensely self-referential (graph) dataset into an RDBMS is a recipe for disaster, however. Many of the performance issues people see with RDBMS seem to stem from this, I believe. If your app is hugely real-time datadriven and the datamining (if necessary) can be offloaded to cron jobs, a K/V store is great. And can scale very quickly and relatively cheaply. If you're doing something hugely relational that, if loaded into a SQL server would require an immense number of self-joins (I'm looking at you, graph analysis) that must be done in real-time and not offloaded to a cron job, a graph database is probably the way to go. They're harder to scale to a huge amount of data, but certain datamining tasks are made much easier - and don't require distributed map-reduce execution. Scaling will get much easier once some systems come out that use k-means (or similar) to cluster and shard the data. That kind of smart scaling would be nigh impossible on either a KV store or a traditional RDBMS. Google gets away with it with BigTable because they can throw so much…

Datomic's immutable storage and time-travel query capabilities are awesome, and I often miss them in other DBs. But Datomic currently isn't designed for write-intensive workloads. And while you can shard Datomic's transactor and then combine multiple DBs in a query (http://nosql.mypopescu.com/post/19310504456/thoughts-about-datomic#com… that's only going to get you so far. However, Apache Geode lets you add custom indexes so it might not be too hard to add Clojure's persistent data structures as a custom index scheme and hook in Apache Geode as a backend to Clojure Datalog: Clojure Datalog: https://github.com/fogus/bacwn Datscript: https://github.com/tonsky/datascript Clojure's Persistent Data Structures for Java: https://github.com/grignaak/clj-ds

Apparently Facebook is not using Cassandra much. For their messaging project they ended up with HBase instead of Cassandra [1]. I don't know if the system replaced the earlier Cassandra system, or complemented it. [1] http://nosql.mypopescu.com/post/1583884165/facebook-the-underlying-tec…

First, your friends with MongoDB are fsync and safe. These are both documented and discussed in more than a few places: http://nosql.mypopescu.com/post/1052759609/mongodb-safe-and-fsync-expl… So, assuming you are writing an ecommerce application, here's where I think these flags come in. - Session data: fsync = true. Wait for a response, and ensure it's written to disk - Internal web analytics: safe = false. Who cares if it's written, I've got an application to serve! - Orders: fsync = true. I know, RDBMS, transactions, blah blah blah. People tend to look at NoSQL and wonder why it doesn't function like MySQL, then they loudly complain how bad the software is. Nobody is writing articles about how Memcached doesn't function like MySQL.

Definitely Web Scale! http://nosql.mypopescu.com/post/1016320617/mongodb-is-web-scale

PostgreSQL: http://nosql.mypopescu.com/post/14114368488/postgresql-hstore-the-key-…

Via http://nosql.mypopescu.com/post/12224364842/mongodb-vs-rdbms-schema-de…

You just don't know about the other side of the coin -- no durability to start with. You'll be flying fast, until one day you realize your data is corrupted. Not saying it is bad thing to optimize for speed, it is just a bit dishonest to downplay the negative effects especially after calling a product a DATA-base. http://nosql.mypopescu.com/post/392868405/mongodb-durability-a-tradeof… http://blog.boxedice.com/2010/02/28/notes-from-a-production-mongodb-de…

Imho this article is too much of an oversimplification. It's forcing artifical labels on some things that don't really fit very well. It's fairly useless if you've already heard about most of these and it's a false start if you haven't. If you do want such a list, a much more accurate and complete one is at http://nosql.mypopescu.com/kb/nosql

Relevant blog post from a few weeks ago that talks about the use cases this code makes possible: http://nosql.mypopescu.com/post/4633402570/riak-zab-zookeepers-zab-pro…

Exactly. Initially they weren't using anything OSS but moved to HAProxy and Redis. They even help update HAProxy from time to time. http://nosql.mypopescu.com/post/2669915777/powered-by-redis-stackoverf…

Currently (mongodb 1.6.x) sharding is a killer feature that might kill yourself. It's just a marketing propaganda. I once tried switch to sharding to avoid high write lock ratio(MongoDB use DB level lock currently, only one write operation allowed at the same time). After sharding, MongoDB even don't know how to count my colleciton. db.mycollection.count() return values at random. mongorestore also failed in sharding setup, there's no error message when I was restoring millions of documents, but after it reported successfully restored, I checked the DB, no documents there. We switched back to more reliable master/slave setup finally. So before trying sharding, you should watch this video http://nosql.mypopescu.com/post/1016320617/mongodb-is-web-scale . It's so true to some extend.

Don't be, their engineers have better things to patch up at the moment than write a detailed post. They will, hopefully, though like foursquare did with their mongodb outage - http://nosql.mypopescu.com/post/1265191137/foursquare-mongodb-outage-p…

Your last point is by far the most important. It looks like one of the goals for Django 1.4 will be more separation between the ORM and relational databases. Alex Gaynor speaks at length about the innards of today's querysets and how he suggests it could be changed to make sense with NoSQL: http://nosql.mypopescu.com/post/904840384/django-and-nosql-databases-r…

For those who may have missed this a couple weeks ago, Twitter also considering Cassandra - http://nosql.mypopescu.com/post/407159447/cassandra-twitter-an-intervi…

I'm at the beginning of a research cycle on graph databases Pretty much the same here. I am not an expert on any of these projects yet, but I can share some of what I've found and been researching so far. and am curious to get some suggestions of your favorite open source projects? Depends on exactly what you are looking for. There are graph "databases" and there are graph "processing engines" and there are subtle differences between them. Some of these things gain massive scalability by running on top of Hadoop, others implement Bulk Synchronous Parallel processing, etc. A few interesting choices you could look at include: Apache Giraph - http://giraph.apache.org/ Titan - http://thinkaurelius.github.io/titan/ Faunus - http://thinkaurelius.github.io/faunus/ Apache Hama - http://hama.apache.org/ - not strictly a graph database OR a graph analytics engine, but a generic BSP platform, written in Java, that works with HDFS. Does include some built in support for Graph processing: http://hama.apache.org/hama_graph_tutorial.html Hypergraphdb - http://www.hypergraphdb.org/index OrientDB - http://www.orientdb.org/ GoldenORB - http://goldenorbos.org/ You may also be interested in Blueprints, which is more of "just an API" but is supported by several of the various graph database projects out there. https://github.com/tinkerpop/blueprints/wiki For what it's worth, I'm leaning towards focusing most of my initial energy on Giraph and/or Hama, but I'm also pretty excited…