Cloudera Blog

From what I heard, while InfluxDB is a great product it may struggle with processing big volumes of data - see https://blog.cloudera.com/benchmarking-time-series-workloads-on-apache…

I learned Hadoop in Grad school in 2013. If you can spend a little bit of cash, get some VMs on AWS, and follow one of the many guides out there (for example, Cloudera) to install Hadoop. Should be enough to build something like: http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hado… I started out trying my VMs on virtualbox, then a couple of different laptops at home, etc, but AWS was the easiest setup in the end.

If your query is in Clojure there are some great tools in addition to those mentioned for data science that leverage Hadoop - in particular - Parkour https://github.com/damballa/parkour/ (Hadoop MapReduce in idiomatic Clojure) If your query is in R then you can convert to a PMML query and then Cascading to Run on Hadoop: http://blog.cloudera.com/blog/2013/11/how-to-use-cascading-pattern-wit…

For most programs, yes they have done their job. However, in certaint categories of applications (for example, server software) it's somewhat of a leaky abstraction. Garbage collector sweeps, circular references, etc are all pains which force you to be aware of how the vm is managing your memory. For an impression, see this excellent blog series on how a certain garbage collector sweep issue was solved: http://blog.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-…

It is also worth noting that Namenode isn't a spof in Hadoop 2. Namenode HA[1] has been in for a while and QJM[1] went in more recently. [1] https://ccp.cloudera.com/display/CDH4DOC/Software+Configuration+for+Sh… [2] http://blog.cloudera.com/blog/2012/10/quorum-based-journaling-in-cdh4-…

Cloudera | Location: San Francisco or Palo Alto, CA | Full-time| Onsite | Software Engineers in Cloud, Test, Site Reliability, Hadoop, Test, Automation, Spark, QA, Security and more | www.cloudera.com Picture yourself at Cloudera in 2016! Our mission: Founded by leading experts on big data from Facebook, Google, Oracle and Yahoo, Cloudera's mission is to bring the power of Hadoop, MapReduce, and distributed storage to companies of all sizes in the enterprise, Internet and government sectors. Over the past few years, Cloudera has experienced tremendous growth, making us the leading contributor to the Hadoop ecosystem and a leading provider of enterprise solutions for Big Data. Cloudera is revolutionizing enterprise data management by offering the first unified Platform for Big Data, an enterprise data hub built on Apache Hadoop™. Cloudera offers enterprises one place to store, process and analyze all their data, empowering them to extend the value of existing investments while enabling fundamental new ways to derive value from their data. Only Cloudera offers everything needed on a journey to an enterprise data hub, including software for business critical data challenges such as storage, access, management, analysis, security and search. Check out all our roles and apply here: http://bit.ly/2016EngCareers Glassdoor's Employees' Choice Awards 2016 http://bit.ly/1ORttqv The Best Startup CEOs To Work For http://read.bi/1KR6zvN 20 Big Data Companies Leading the Way…

We were just joking at work today how Cloudera's new Kudu "fast storage system"[1] compares itself to "fast" HBase. At least there they were talking about random access (something that HBase is reasonably ok at), and put HDFS on the "slow" end of the spectrum when talking about storage. This system claims: Experiments show that WTF can qualitatively outperform the industry-standard HDFS distributed filesystem, up to a factor of four in a sorting benchmark, by reducing I/O costs They benchmark a sort 300GB of data. You know what's fast at sorting 300GB data? My laptop. Sort a PB[2] much faster than Spark+HDFS and then it might be worth looking at. Even 100TB[3] is an interesting size. Maybe I'm being too harsh? What am I missing here? [1] http://blog.cloudera.com/blog/2015/09/kudu-new-apache-hadoop-storage-f… [2] https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html (note that this was on disk with HDFS, not Spark in memory). [3] http://sortbenchmark.org/

Although I have a lot of respect for the amplab, they did not do their due diligence with that benchmark. Mainly for a few reasons, they didn't test using columnar storage in Hadoop (ORC / Parquet), which is what Redshift is using underneath (a proprietary columnar store). Also, the most complicated query they ran was a two table join, and from what I can tell, there wasn't any concurrent workload testing. (disclaimer: I'm a Cloudera employee): I recommend checking out the following blog, not because my employer wrote it, but because the guys behind the benchmark did an incredible job making the benchmark competitive. They also show metrics that a lot of the other people are not showing, for example concurrent workload capabilities, CPU efficiency, etc. Impala, Hive (on Tez), Shark, Presto http://blog.cloudera.com/blog/2014/05/new-sql-choices-in-the-apache-ha…

S3 ListObjects calls are eventually consistent (i.e. list-after-put). EMRFS [1] and S3Guard [2] mitigate this for data processing use cases. [1] - https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-fs.html [2] - https://blog.cloudera.com/introducing-s3guard-s3-consistency-for-apach…

SMM is another option and has some really cool features for alerting as well. Runs in any cloud or on premise. https://blog.cloudera.com/smm-1-2-released-with-powerful-new-alerting-…

1) Put it on figshare or github. Your website might be dead next year. Those services have a better chance of survival. 2) CSV, JSON works fine if your dataset is just a few numbers and strings. GitHub will preview csv files as HTML tables. 3) If you need the efficiency of binary data and more robust data containers I would look into - Parquet https://parquet.apache.org/documentation/latest/ and - Arvo http://avro.apache.org/ http://blog.cloudera.com/blog/2009/11/avro-a-new-format-for-data-inter… 4) DataScientists work with R/Pandas "DataFrames". If you are familiar with either one, import the data into a data frame and use an export method to do the serialization for you: https://pandas.pydata.org/pandas-docs/stable/api.html#id12

For those curious about background: https://blog.cloudera.com/blog/2013/02/inside-cloudera-impala-runtime-…

Check out Apache Arrow and Feather for a "local" equivalent of Parquet https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-f…

It's not really a service at all, it is a in-memory data format intended to be shareable between processes. The project also includes libraries for C++, Java and Python. This post explains the intention better than the project webpage: http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fast-…

More technical details available here: http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fast-…

Note that this proposal is being back not only by Google, but also Cloudera, Data Artisans, Talend, Cask, PayPal, ... Some other posts on the announcement: http://googlecloudplatform.blogspot.com/2016/01/Dataflow-and-open-sour… http://blog.cloudera.com/blog/2016/01/spark-dataflow-joins-googles-dat… http://data-artisans.com/dataflow-proposed-as-apache-incubator-project… http://blog.cask.co/2016/01/cask-anticipates-googles-dataflow-to-flour…

Makes most sense to compare Impala and Spark architecturally. Ibis will eventually integrate with Spark. We've been focusing on Impala integration for reasons cited here: http://blog.cloudera.com/blog/2015/07/getting-started-with-ibis-and-ho… In particular, we're working on byte-level shared-memory integration with Impala (which is implemented in C++ with LLVM runtime codegen — the project's tech lead, Marcel Kornacker, was the tech lead for Google F1's query engine) to run user-defined logic without data serialization / memory usage overhead. This also opens up Python's HPC / scientific computing stack and existing data libraries to be run in a Hadoop setting without Python-JVM interoperability issues.

Cloudera now includes Spark... http://vision.cloudera.com/mapreduce-spark/ http://blog.cloudera.com/blog/category/spark/

Subscribe to Data Science Weekly: http://www.datascienceweekly.org/ http://blog.zipfianacademy.com/ http://blog.cloudera.com/blog/category/data-science/ http://www.hilarymason.com/ http://mathbabe.org/ http://fivethirtyeight.blogs.nytimes.com/ http://blog.kaggle.com/ http://grepalex.com/ http://pulseblog.emc.com/category/big-data/ http://radar.oreilly.com/ http://flowingdata.com/ http://oreilly.com/data/newsletter.html http://www.gapminder.org/ http://mlcomp.org/

Some interesting use cases are also described on Cloudera's developer blog, at http://blog.cloudera.com/blog/2013/11/putting-spark-to-use-fast-in-mem…

There is no mechanism for search "completeness" without access to the fire hose. http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hado… ^ that is a good resource if you ever get access to the firehose. Shoot me an email [email protected], and when I get home I can shoot you an email with some example code that Ive written in the past.

CAP theorem can apply to any clustered system, it doesn't have to be multi-site. What happens if 6 of your 12 machines die? What if they get cut off from the other 6? edit: There's a bit of discussion further down about the SQL implementation. That's something I was very curious about as well. The projects linked below spend a lot of time working on supporting full ANSI SQL, and reducing latency by pushing down as many operations as possible. The Overview page doesn't appear to mention how filtering, aggregation, windowing, etc. work in your system. Also, I noticed on your website that you compare InfiniSQL to Hadoop. How do you feel it compares to Impala (http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-querie… and Shark (https://amplab.cs.berkeley.edu/projects/shark-making-apache-hive-run-a…

The author is missing a big gap between 5TB - 1PB. For most workloads, I would not look to Hadoop at the 5TB+ scale of data. I would first look at Impala or Redshift. http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-querie…

Most of gcc is still written in C. Only a tiny amount of new code is in C++. In my opinion, that's the worst of both worlds-- inconsistent code, plus C++ overheads. I work with some fairly recent version of both compilers-- gcc 4.7.2 and clang 3.2. I also know a developer on the LLVM team. clang is way ahead of gcc in many ways, and the gcc devs know it. We still ship software built with gcc, rather than clang, because gcc is still better at optimization. (gcc also supports more obscure targets, though that's not our biggest concern). We also make use of llvm's ability to dynamically recompile code on the fly in Cloudera Impala. More information here: http://blog.cloudera.com/blog/2013/02/inside-cloudera-impala-runtime-c… This would not be possible with gcc no matter how much work we did (license issues-- Impala is Apache 2.0, gcc is GPLv3).

(I wrote MoSQL) PostgreSQL scales surprisingly well for this purpose, and is much nicer for interactive queries than Hadoop/Hive. We use Impala[1] for some larger datasets, but Impala is comparatively new, and it's nice to have something as battle-tested as postgres here. As for the "why do we need realtime?": In my mind the benefit of a near-realtime replica is not that you actually often need it, but that it means you never have to ask the question of "Was this snapshot refreshed recently enough?", and never end up having to wait several hours for an enormous dump/load operation, when you realize you did need newer data. [1] http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-querie…

A link to the project page for Sqoop - http://sqoop.apache.org/index.html A link to an overview of Sqoop from Cloudera http://blog.cloudera.com/blog/2011/10/apache-sqoop-overview/

I noted that Marcel Kornacker, the architect of Impala, was the lead developer for the query engine of Google’s F1 project. In the end of the article: http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-querie…

"Cloudera Impala: Real-Time Queries in Apache Hadoop, For Real" http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-querie… Impala source code: https://github.com/cloudera/impala Impala documentation can be found here: https://ccp.cloudera.com/display/IMPALA10BETADOC/Cloudera+Impala+1.0+B…

The use cases for such random access was from our experiences to maintain large-scale training data, with the needs to debugging model performance against to data, which requires fast slicing and dicing over the dataset, filtering into subset of dataset that satisfy certain distributions, and visualize them on internal debugging UI interactively. A few of users use random access to do shuffling and training on subsets as well. Many of them migrated from a system where you can store asset URL in your format of choice (say parquet/tfrecord), to Lance, partially because putting asset (i.e., image, large tensors, lidar point cloud) physically together can lead to better scan perf than loading a lot of small files from S3 or on-prem object store / file systems, due to much less metadata server load over the directory / key&value structures. (i.e. Some of our users see similar issues like this decade old article https://blog.cloudera.com/the-small-files-problem/) One motivation to design lance is to avoid creating a new copy of training dataset just for one model / training iteration. This is one copy of data for maintenance (update/schema evolution/deletion), analytics & visualization, and training. For scanning, Lance has been proved slightly faster than Tfrecord and parquet over object store (S3 and GCS). I'd contribute the scan performance just to Rust and tokio async I/O. It is not necessary a better design in scan as we just try to scan "no more data than parquet" when…

Database optimization posts are always interesting, but it's really hard to do any apples to apples comparison. Your performance is going to depend mostly on your hardware, internal database settings and tunings, and OS level tunings. I'm glad this one included some insight into the SQLite settings disabled, but there's always going to be too many factors to easily compare this to your own setup. For most SQL systems, the fastest way to do inserts is always just going to batched inserts. There's maybe some extra tricks to reduce network costs/optimize batches [0], but at it's core you are still essentially inserting into the table through the normal insert path. You can basically then only try and reduce the amount of work done on the DB side per insert, or optimize your OS for your workload. Some other DB systems (more common in NoSQL) let you actually do real bulk loads [1] where you are writing direct(ish) database files and actually bypassing much of the normal write path. [0] https://dev.mysql.com/doc/refman/5.7/en/insert-optimization.html [1] https://blog.cloudera.com/how-to-use-hbase-bulk-loading-and-why/