Michael Noll

Michael Noll has some really good introductory articles to Big Data, and I highly recommend his posts for anyone starting in this field. His post on writing Python streaming programs in Hadoop - http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-prog… - got me started in this area two years ago, and that's what I'm still doing these days! :)

There isn't any pain. Best of all it's easy to setup hadoop on a single machine, or two machines here and there and then them into a large cluster: Here's a good tutorial that may help: http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Sin… (If you're using a Linux machine just follow that; otherwise get VMWare running Ubuntu and set it up there).

I found this article to be cool https://www.michael-noll.com/blog/2013/12/02/twitter-algebird-monoid-m…

Exactly, which is why the expert information is teased out in a couple in depth blog posts. https://www.michael-noll.com/projects/ Does a very good job in this regard. I think if you have a consulting agency where you can train or lead on multiple projects, then a book could be a very good sales vehicle.

Having written MR jobs in the original Java format, I can tell you that it's quite valuable to do so before learning a DSL like Scalding or Pig. I found http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-s… to be a very good tutorial for a quickstart guide to Hadoop.

have you tried have you tried http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-s… and then migrate to multi node http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-m…

Is it this one :) ? http://www.michael-noll.com/projects/spear-algorithm/

It would be really nice to have dataset available for researchers with username, their posts and their comments. As an example, we could use the "SPEAR Algorithm"[1] to find expertise in specific domains per HN user. [1] http://www.michael-noll.com/projects/spear-algorithm/

This python API into Delicious could give you a place to start: http://www.michael-noll.com/projects/delicious-python-api/ Unfortunately Delicious will throttle you if you hit the service more often than once a second so you might not be able to get too much valuable information.

The Hadoop book gives lots of insights into the internals of Hadoop. But if you want to get started quickly you might be better off just reading a quick guide (http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Sin… and using the Pig console.

Python: http://www.michael-noll.com/wiki/Writing_An_Hadoop_MapReduce_Program_I… Ruby: http://blog.rapleaf.com/dev/?p=16 It's pretty hokey, either running through the shell or calling a REST service through a proxy, but it's been done if you're allergic to non-scripting languages.

Here's a note about using Hadoop in Python: http://www.michael-noll.com/wiki/Writing_An_Hadoop_MapReduce_Program_I…

Hadoop itself consists of different parts: HDFS is a distributed filesystem that can span across lot's of machines and stores data in blobs of varying sizes. MapReduce is the Google idea from before 2004 how to do calculations on lot's of data. Now there is also YARN that could be described as a general job scheduler. At the moment a lot of people use software on YARK like Spark (does more in memory, is faster, can use the GPU on the cluster machines). So if you have biological data you could feed that into a HDFS and have Spark or MapReduce jobs that process the data. The clue about Hadoop is that don't need to care about getting the clustering and distributed setup right. This is done by Hadoop for you. You program like you would program a single thread algorithm (at least in the simple cases). E.g. this Python code counts words over as much data as you want in parallel: http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-prog… If you google each of these projects you'll find a lot of information. Here are the original papers that should give a good idea: - HFDS (Google Filesystem was the original idea - Hadoop was a free implementation by Yahoo - http://static.googleusercontent.com/media/research.google.com/en//arch… - MapReduce - http://static.googleusercontent.com/media/research.google.com/en//arch… - Spark: http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf If you want a good…

Java's not actually a bad choice, considering some of the tools that are available. Hadoop[1], HBase[2], etc. Depending on what you are trying to accomplish of course, you could also include Lucene[3], Solr[4], Mahout[5] and Weka[6] in that list. Of course cool Java libraries don't mean you have to work in Java. Scala[7] seems to be gaining quite a bit of momentum, as does Clojure[8]. From your reading list, it sounds like you may be interested in predictive analysis, machine learning, etc. If so, Python[9] is pretty popular in those circles and has some nice libraries available as well[10]. And you can do neat things like use a Hadoop cluster for map/reduce processing, even if you have a client written in Python (or whatever)[11]. Of course Map/Reduce[12] isn't the only distributed processing model out there. You might want to learn MPI[13] then explore the MPI bindings that are available for many (most?) languages, and look at other models of distributed / parallel computing. Anyway, to summarize, I'd say that if you already know one or more of Java, Python, C, C++ or Scala, you're in good shape. If not, some combination of the above should fit the bill. And yes, R might be handy as well. Not to take anything away from Erlang, Haskell, Forth, Factor, Scheme, Lisp, OCaml, Perl, or any other language of course. I'm just not as familiar with those so I can't say much about them. 1. http://hadoop.apache.org/core/ 2. http://hbase.apache.org/ 3.…

What If We Could Rebuild Kafka From Scratch? - Gunnar Morling (https://news.ycombinator.com/item?id=43790420) 1. Five Advantages of Log-Based Change Data Capture (https://debezium.io/blog/2018/07/19/advantages-of-log-based-change-dat… 2. Why Data Architecture Should Guide Your Query Engine Decision (https://www.onehouse.ai/blog/why-data-architecture-should-guide-your-q… 3. Reimagine Apache Kafka (https://streamnative.io/blog/ursa-reimagine-apache-kafka-for-the-cost-… 4. Book Review: Designing Data-Intensive Applications | Henrik Warne's blog (https://henrikwarne.com/2019/07/27/book-review-designing-data-intensiv… 5. Of Streams and Tables in Kafka and Stream Processing, Part 1 (https://www.michael-noll.com/blog/2018/04/05/of-stream-and-tables-in-k…

There're many things you can do with a spare linux box. I do keep a few 'project ideas' links just in case I got some spare time on hand. Home file server with ZFS - http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/ Proxy Server - http://devcentral.f5.com/weblogs/macvittie/archive/2008/10/08/8-things… Map-Reduce fun with Hadoop - http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Singl… Donate bandwidth with a Tor Server - http://ubuntu-tutorials.com/2007/08/31/how-to-run-a-tor-server-donate-… More good ideas - http://www.dailycupoftech.com/2007/09/17/put-that-old-computer-to-good… A Dial-In server - http://www.howtoforge.com/linux_dialin_server