DBMS2

It's for companies that have large data warehouse analytics systems. -- highlights: "Syncsort cites its main data integration competitors as being, no surprise, Ab Initio (their favorite choice to compare themselves to), Informatica (with PowerCenter), and IBM (with DataStage). Syncsort DMExpress runs on dedicated boxes. Unlike most of its competitors, this is not a real-time product it is a batch product that you schedule. Syncsort also told a story of an unnamed customer for whom Oracle utterly choked on joining 5 tables of 1 terabyte each. (27 days to run with clever workarounds.) DMExpress did the join in 6 hours." -- So, it's apparently so large and enterprisey that getting it to work right would take a large amount of effort, a sizable budget, and paid tech support. https://www.dbms2.com/2011/04/07/introduction-to-syncsort-and-dmexpres…

On the new age tech, these may help: https://hadoopweekly.com/ http://www.wildml.com/newsletter/ https://thomaswdinsmore.com/ http://www.dbms2.com/

Classic marketing pitch from a little company that wants to claim it's much more significant than it is: 1. Claim a must-have set of requirements that ... 2. ... happen to match its product's feature set ... 3. ... but not its competitors.' http://slamdata.com/whitepapers/characteristics-of-nosql-analytics-sys… is presumably the core of the argument. I tend not to pay attention to such claims until the company rephrases them more honestly. That said, a brief discussion of what is really happening is in http://www.dbms2.com/2015/09/10/mongodb-update/ Would more be better? Sure.

If you look up columnar databases, you can find a whole host of research on how these things store data. Storing values column by column to do vector processing is pretty common, but I can't recall seeing one built for a GPU before. Kudos to the dev for diving in to implementing one, it's pretty neat stuff. Try finding research papers about MonetDB, Vertica/C-Store, or Vectorwise for some background. Or follow the links here: http://www.dbms2.com/category/database-theory-practice/columnar-databa…

People seem to be confused about what IBM's System S actually is. For a more technical look at their new offering I suggest these two posts: http://www.dbms2.com/2009/05/13/ibm-system-s-infosphere-streams-proces… http://www.dbms2.com/2009/05/18/followup-on-ibm-system-sinfosphere-str… Quick summary: It's for doing complex event processing of large streams of data. Think financial data or any other such sort of never ending flow of massive amounts of data that you would need to process very quickly to identify patterns and for other analysis such as determining when you hit certain thresholds or trends. It's supposed to be very flexible so that it can act on a large amount of unstructured data as well as very scalable. It defines its own processing language too.

The cardinal rule of databases: http://www.dbms2.com/2013/03/18/dbms-development-marklogic-hadoop/ Ignoring that leads to long postmortems about how your embedded db failed in a way you didn’t expect: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2… SQLite is good Rocksdb is good You don’t need pure go.

CMU's Andy Pavlo is probably at the forefront in this area, here's what they were working on (they're re-doing it all over again now as a yet to be announced project): https://news.ycombinator.com/item?id=14391523 Also, I'd encourage you to go through their Advanced Database class: https://www.youtube.com/playlist?list=PLSE8ODhjZXjYutVzTeAds8xUt1rcmyT… Michael Stonebreaker, a Turing awardee, called for complete destruction of old database order in 2008: https://www.dbms2.com/2008/02/18/mike-stonebraker-calls-for-the-comple… post which, I think, they created VoltDB (H-Store) and SciDB. Interesting to note that Andy Pavlo worked on Stonebreaker's H-Store.

IMO, it's very promising but still extremely immature. I still believe in Curt Monash's cardinal rule of RDBMS development: http://www.dbms2.com/2013/03/18/dbms-development-marklogic-hadoop/

> Oracle has gone into the arena of companies like IBM and CA ... Exactly: http://www.dbms2.com/2015/12/31/oracle-as-the-new-ibm-has-a-long-decli… Personal story - they're pushing exadata and exalytics. Products based on 10 year products based on 15+ year old ideas.

I'm not sure what GP means by Ross "disowning the entire R project". Maybe it's a brain fart. But Ross does (if this is indeed him commenting on dbms2.com) want people to not use Revolution Analytics fork: http://www.dbms2.com/2010/05/04/revolution-analytics-confused/

Salesforce does this: http://www.dbms2.com/2011/09/15/database-architecture-salesforce-com-f…

I'm a little blown away by the total lack of technical understanding when it comes to MongoDB. This person is obviously very technical, so why would he have chosen MongoDB in the first place? It isn't like MongoDb's technical shortcomings are a secret. The description of how MongoDb does sharding and distributed queries should have immediately raised red flags to any who has even a modicum of CAP understanding. If it sounds complex, a total hack and impossible to manage it probably is. For gosh sakes, even the 10gen guys admit MongoDB lost data a year(!) ago. [1] If a database lost data in a single server configuration, why would you trust it as a cluster? 1: http://www.dbms2.com/2011/04/04/the-mongodb-story/

"Hadoop futures and enhancements" http://www.dbms2.com/2011/07/10/hadoop-futures-and-enhancements/ Monash Research / Hadoop news : http://www.dbms2.com/category/products-and-vendors/hadoop-products-and…

From http://www.hadapt.com/about (very slow to load right now), two of the guys are Daniel Abadi and Kamil Bajda-Pawlikowski. They worked on HadoopDB [1], which was an attempt at creating a parallel DB on top of Hadoop, that would leverage mapreduce scale, while taking advantage of the benefits of relational databases. I think the hope was to have some sort of system capable of taking on Vertica. Looks like they've turned it into a product. Ah, more coverage here [2]. 1: http://db.cs.yale.edu/hadoopdb/hadoopdb.html 2: http://www.dbms2.com/2011/03/23/hadapt-commercialized-hadoopdb/

For a similar but more mature use-case see this australian company described in a recent Curt Monash post: http://www.dbms2.com/2010/06/19/objectivity-infinite-graph/ (I don't know more about them, but this is one possible model for us.)

...and just a few points-of-sale. The Wal-Mart data warehouse operation is legendary in the industry. As of 2008, they were running Teradata, w/ 2.5 PB of data: http://www.dbms2.com/2008/10/15/teradatas-petabyte-power-players/ By comparison, Facebook had ~2.5 PB in Hadoop/Hive in early 2009: http://www.dbms2.com/2009/05/11/facebook-hadoop-and-hive/

databases are for live queries No, databases are also commonly used for batch queries. Are you familiar with data warehousing? Teradata, Greenplum, Netezza, and similar products? This sort of stuff, for example: http://www.dbms2.com/2009/04/30/ebays-two-enormous-data-warehouses/ Parallel databases have been around since the late 1980s, and can be used for many of the same things MapReduce is often used for (large-scale batch data analysis). Hence the comparison.

Curt Monash's take on it :- http://www.dbms2.com/2008/02/18/mike-stonebraker-calls-for-the-complet…

Agree. & this makes sense from an architectural perspective, since the demands of each domain are different. I was arguing from practical terms and my own experience. Even in established, DB-heavy, "we have hundreds of DBAs" organisations, you can't have a porcelain schema. It evolves over time, breaks, get optimised and de-optimised. The Oracle shops I've worked in don't have one schema, they'll have dozens. Usually because they can't shoehorn emerging requirements into the databases they have. Or because of project timelines. Or they need to be performance tuned in some special way, which in the case of Oracle could be considered a dark art. These orgs may have just one big data warehouse, but that becomes a dumping ground. It's possible to do your dimensional modelling (I worked on star schemas in the past, but that's about it) -- but it's a big ball of mud to try and weave together. Even then your OLAP is under-stress because it's got a lot of ground to cover. Usually they start out as marketing engines, but get co-opted into all sorts of things. The most common and troublesome is regulatory reporting. Once your data warehouse gets used for something like regulatory reporting you're a bit stuck as nobody wants to touch it. Then, because point to point integrations between the databases is brittle and cost-prohibitive -- they break the golden rule and take data out of the OLAP and put it back into the OLTP. I won't name names, but this is commonplace. SalesForce is…

This? http://www.dbms2.com/2005/11/12/transrelational-23/

First off - whatever you're doing, you'll probably be fine using either. That said, I'm generally more frustrated when using MySQL than when using PG. Here's a sample of the problems I've encountered from using both MySQL and PG. I haven't updated my list in a while now - please feel free to correct me on things - but hopefully it's a little more illustrative than that Wikipedia feature matrix, and a little more specific to MySQL vs. PG. (This list is a cleaned-up selection from my notes wiki at http://yz.mit.edu/notes/Hackery.) No referential integrity. No constraints (CHECK). No sort merge join, let alone hash-join. http://www.dbms2.com/2008/07/10/how-is-mysqls-join-performance-these-d… http://www.mysqlperformanceblog.com/2006/06/09/why-mysql-could-be-slow… Generally poor at analytical workloads, since it's designed for transactional workloads. Can't reopen TEMP table - WTF? (Still not fixed!) http://bugs.mysql.com/bug.php?id=10327 Multiple storage engines has always restricted progress: http://www.mysqlperformanceblog.com/2010/05/08/the-doom-of-multiple-st… (PG also supported multiple storage engines in 80s, then concentrated on one) No WITH clause: http://stackoverflow.com/questions/324935/mysql-with-clause Crappy errors: “Incorrect key file for table ‘stock’; try to repair it” on “alter table stock add constraint pk_stock primary key (s_w_id, s_i_id);” where stock is in InnoDB (which has no “repair table”) means I have no…