AMPLab Blog

It's fascinating to see the descendants of the original AMP Lab (https://amplab.cs.berkeley.edu/) are still generating value long after it ended. For reference, Spark and Mesos are two other projects that came out of the same effort. This blog post from the authors https://medium.com/@zongheng_yang/skypilot-ml-and-data-science-on-any-… I believe autostop (simply stopping idle clusters) alone can masively (X times) costs for many organizations.

One of the authors here! We are glad you like it! The project is indeed driven by the authors listed on the paper and also the knowledge and experience that was accumulated in the AMPLab (the predecessor of the RISELab, see https://amplab.cs.berkeley.edu/). If you look at the github history, we've been working on it for longer than a year and had various prototypes before that, so it doesn't come out of "thin air" ;) The lab's sponsors are also helpful, some of them have been experimenting with the system internally and giving us feedback.

> I would love to see MR and Spark compete on the exact same hardware configuration. You may find this benchmark [1] interesting to read. It needs some updating (a lot has changed since February 2014), but it compares Shark (which uses Spark as its execution engine) to Hive (using Hadoop 1 MapReduce as its execution engine) and a number of other systems. The benchmark is run on EC2 and is detailed in such a way that it should be independently verifiable. Hive and Shark are run on identically sized clusters, though I don't know if the other details of the configuration were identical. [1] https://amplab.cs.berkeley.edu/benchmark/

Note that the suggested benchmark (https://amplab.cs.berkeley.edu/benchmark/) is a slightly modified version of the Hive Benchmark. Both of these are just 3-4 tables total and 4 very basic queries. I recommend looking at something more realistic (e.g. TPC-DS, TPC-H, etc).

Thanks for the clarifications, I wasn't not meaning to discredit your article in anyway. I am just trying to help people understand that these databases are very different from eachother, and were created to solve different use cases. > Hadoop works great for "big data" (petabytes) but it was overkill for us (working with terabytes). This is a very common misunderstanding about Hadoop, terabytes of data is still very large when you want to apply complex transformations, or allow someone to run queries over it with low latency expectations (a few seconds). Impala does not use MapReduce, it was designed with low latency goals in mind...speeds are comparable to RedShift (https://amplab.cs.berkeley.edu/benchmark/), although they are citing an older version.

Note: the underlying comparison to other systems is from a 2014 blogpost [1] which suggest they used the m2.4xlarge series of EC2 VMs (which were Nehalem class parts from 2010). Nehalem vs Haswell or Broadwell (the likely parts underlying Lambda) is a pretty big jump. Disclosure: I work on Google Cloud, but I'm just pointing out a fact ;). [1] https://amplab.cs.berkeley.edu/benchmark/

In case you don't know who David Patterson is: * He pioneered RISC, and coined the term. About all smartphones are ARM-based * He came up with RAID, coined the term. * He cofounded Berkeley's AMPLab, which is the birthplace of Spark. * Among other AMPLab activities he has are SNAP; It is about DNA analysis with the goal of helping solve cancer (which is an acquired genetic disease). In 2014 this project helped save a live [1] [1] https://amplab.cs.berkeley.edu/snap-helps-save-a-life/

It was, but judging from Mike's comments [1], he was also being a bit naïf coming in expecting otherwise. This kind of stuff happens often enough there are several common sense guides/books for "talking to the media" (see eg [2]). If you are in academia, you might definitely want to familiarize yourself with this stuff at some point. [1] https://amplab.cs.berkeley.edu/2014/10/22/big-data-hype-the-media-and-… [2] http://www.ucsusa.org/sites/default/files/legacy/assets/documents/glob…

Also of interest is Michael Jordan's response to the original article with him: https://amplab.cs.berkeley.edu/2014/10/22/big-data-hype-the-media-and-…

I think Spark is from the Berkeley AMPLabs which is primarily focused of machine intelligence, data mining and artificial intelligence in general. Most, if not all, of there projects (https://amplab.cs.berkeley.edu/projects/) are geared at solving similar problems. I just don't see how this particular application is so important. At most, it seems to be demonstration of sparks capacities but again it was geared for high performance cluster computing anyways.

They reference GraphX (http://amplab.github.io/graphx/). GraphX is essentially GraphLab built on Spark, and the GraphX project lead is Joey Gonzalez (https://amplab.cs.berkeley.edu/author/jegonzal/), the creator of GraphLab.

It's unlikely to be much different than the recent Berkeley benchmarks, since they all ran on AWS: https://amplab.cs.berkeley.edu/benchmark/

>>Show me the numbers https://amplab.cs.berkeley.edu/benchmark/

CAP theorem can apply to any clustered system, it doesn't have to be multi-site. What happens if 6 of your 12 machines die? What if they get cut off from the other 6? edit: There's a bit of discussion further down about the SQL implementation. That's something I was very curious about as well. The projects linked below spend a lot of time working on supporting full ANSI SQL, and reducing latency by pushing down as many operations as possible. The Overview page doesn't appear to mention how filtering, aggregation, windowing, etc. work in your system. Also, I noticed on your website that you compare InfiniSQL to Hadoop. How do you feel it compares to Impala (http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-querie… and Shark (https://amplab.cs.berkeley.edu/projects/shark-making-apache-hive-run-a…

In addition to Google... DataBricks is a multibillion dollar company and was based on research at the AMPLab UC Berkeley, which was funded by an NSF Expeditions grant. https://amplab.cs.berkeley.edu/news/amp-awarded-10m-nsf-expeditions-gr… Duolingo is a multibillion dollar company and was based on NSF funded research. https://www.nsf.gov/science-matters/nsf-gave-duolingo-its-wings My previous startup, Wombat Security Technologies, was based on NSF funded research of just $1.2M. It led to over 200 jobs, and we helped protect millions of people around the world through our cybersecurity training. I'm sure there are dozens of other startups that I don't know of also based on US Federal funding from NSF and NIH. Strong science leads to a strong economy, and a strong economy is essential to national security. We're also in an AI arms race with other countries around the world. Cutting science funding right now is a massive self-inflicted wound. For everyone who is a US citizen, please write your Senators and House representatives pushing back against the chaos and the proposed science cuts. It only takes a few minutes, and the future of science in the US needs every bit of help it can get right now.

Thank you for responding. I indeed spent time/effort in way of my crafted reply because it caught me eye. I haven't fully parsed the paper but covered a number of pages that colored the nature of my inquiry. I in no way intended to take anything away from the author of the paper but wanted to get at what you yourself declared : "was accumulated in the AMPLab (the predecessor of the RISELab, see https://amplab.cs.berkeley.edu/)" as The backstory behind the paper as I clearly surmised there was one its historical nature. I also wanted to understand how long this was being worked due to the nature of the language used in the paper and how the concepts and language familiarly fit in with other things I've seen. And this right here : "The lab's sponsors are also helpful, some of them have been experimenting with the system internally and giving us feedback." Yes, I understand the nature of this is moreso for corporate use cases than it is for academic and furthering therein. I was in search of names but already have a number of them I can surmise and a handful more that I will derive. I think its interesting what is being done here but there were choice words that were stated in the paper that limit it. At this juncture and time in the state of AI development, I will reserve any other commentary beyond stating that there are an incredible amount of fundamental limits in approaching things this way that fall on deaf ears due to the shut off nature/sponsorship of such…

I agree with your findings on Hadoop. It is a very complicated ecosystem. The core is tied to JVM and abstractions of map-reduce are heavy and difficult to use. There are multiple domain specific languages/api to do the same thing : pig, cascading. Yet if you pick one you cant use other. Hadoop feels like tower of babel. But I am not in the favour of creating another stack as Hadoop replacement. Spark fills the role of distributed computation framework very well. It uses scala a functional language which provides neat abstractions for map,reduce,filter and other operations. It introduces RDD to abstract data. All the components graphx, sparksql, blinkdb use scala. Also building a new map-reduce engine is a difficult task. The computation engine will lie in the middle of stack not on top as shown. Libraries will sit on top of that, so if you build an engine you will have to build everything on top of that. See the Spark ecosystem : https://amplab.cs.berkeley.edu/software/

My list is somewhat of the same but also different as well. It is focused on specialization within those trends rather than just a broad overview. The difference is specialization or niche in these markets can mean the difference from surviving as a business or failing. Something else to be aware of is if you develop something now you are most likely to see it's production use within 3-5 years of development and then you are most likely to see it's market either growing or not in 10 years. This is an interesting dichotomy because it means picking things that may be hard but boring, or fun depending on how you look at them. I am not affiliated with any of these companies. I just think some of them are well-placed in the market. -Devops (mostly containers and virtualization software, but also fast/easy build software, customizable configs that adapt better and searchable/informative apis) -Devops massive ssl/managing keys/passwords deployment software (there are a few automated tools that may help with this but there is in general not too many companies currently handling it) -Devops massive worker model software or software that helps automation in newer/interesting ways like stackengine (http://stackengine.com/blog/) just got $1 million in funding and provides a way to convert vagrant images to docker images and back again in a fast fashion -Devops with companies that provide continuous integration at scale or preconfigured devops things that require too much learning…

Redshift is an especially limited SQL engine considering it doesn't support UDFs. It is wicked fast, but what you get in speed you lose in flexibility. Current (well, February, but fairly current) benchmarks[0] place Impala and Shark (SQL on top of Spark) within grasp of Redshift while pulling data from disk and, for certain workloads, on par or faster than Redshift. This is without using a columnar file format. Impala is impressive technology, but it does require you to run dedicated Impala daemons as it doesn't use map reduce under the hood. Shark is especially interesting, however, because it is fast AND build on top of spark, so you can run raw Spark jobs, SQL queries, graph processing and ML all on the same cluster. Shark currently uses Hive to generate it's query plans, but the Spark project is working on implementing it's own SQL engine called Catalyst[1] that promises to be a significant improvement. [0] https://amplab.cs.berkeley.edu/benchmark/ [1] https://spark-summit.org/talk/armbrust-catalyst-a-query-optimization-f…