Nathan Marz

Reminds me of Nathan Marz's great piece about lessons learnt from creating, and maintaining Storm as OSS: http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned…

Nathan Marz (of Apache Storm fame) also wrote about this topic recently: "Now, I don't want to comment on the effectiveness of open floor plans for fields other than my own. But for software development, this is the single best way to sabotage the productivity of your entire engineering team." http://nathanmarz.com/blog/the-inexplicable-rise-of-open-floor-plans-i…

Sawzall Language spec: http://szl.googlecode.com/svn/doc/sawzall-spec.html You might also want to take a look at Cascalog. It offers a much nicer API — just list the various predicates and it'll take care of the rest for you, e.g. to find all the guys following Emily: (follows "emily" ?person) (gender ?person "m") http://nathanmarz.com/blog/introducing-cascalog-a-clojure-based-query-…

The Delancey Street Foundation is an organization that has recuperated thousands of people who have hit rock bottom. And they do it without any government funding (they're self-supporting). One of the best examples of social entrepreneurship out there. http://www.delanceystreetfoundation.org/ And if you're interested in the details of how the organization operates, I did some research and wrote about the organization on my blog: http://nathanmarz.com/blog/mimi-silbert-the-greatest-hacker-in-the-wor…

It is interesting to read the history of Storm from the original author (Nathan Marz) himself: http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned… Clojure allowed him to build the initial Storm release by himself in only 5 months. > I made all of Storm's APIs in Java, but implemented Storm in Clojure. By keeping Storm's APIs 100% Java, Storm was ensured to have a very large amount of potential users. By doing the implementation in Clojure, I was able to be a lot more productive and get the project working sooner. Also interesting is that Nathan has now founded a startup working on something which sounds like the next evolution of Storm, and he's building it in Clojure again: https://redplanetlabs.com/ They recently got 5M in funding and are hiring. Building a Clojure team for it. Personally, I feel the Storm 2.0 release rewrite to Java is really just a case of the new maintainers not knowing Clojure very well, and being primarily Java devs, looking for contributions from people using Storm, which tend itself to be mostly Java shops. Clojure is not quick to pick up, and very different from Java, so unlike Scala, Kotlin, C#, Go, etc. Someone with a Java background and no Clojure or Lisp experience won't be able to contribute as easily. And that's fine, and possibly best for Storm's future now that Nathan has moved on. Unless a Clojure dev had stepped up and adopted Storm, this change isn't at all surprising.

It's hard to compare Clojure and Scala. Scala is a multi-paradigm programming language with strong OOP support and functional support. It's arguably more verbose than Clojure but looks much more similar to Java. Clojure is a Lisp. Lisp stands aside all other programming languages, first of all, because it supports syntactic abstraction (a.k.a. "code is data"). Hardcode addicts (I'm not one of them) say there are only two programming languages - Lisp and non-Lisp. Here's a good comparison of Scala and Clojure: http://stackoverflow.com/questions/1314732/scala-vs-groovy-vs-clojure When we made the decision to switch to Clojure, several things affected it, in no particular order: - we had some people who were already very proficient in Lisp - we liked how expressive and compact it was - Lisp is considered to possess immense expressive power (see http://www.paulgraham.com/lisp.html) - we were enamoured by Cascalog (http://nathanmarz.com/blog/introducing-cascalog-a-clojure-based-query-… and it's written in and for Clojure. This one payed off very well. - Lisp has a reputation of being great at manipulating data: lists, graphs, etc. Here's a good answer from one of our engineers: http://www.quora.com/Clojure/Why-would-someone-learn-Clojure As for libraries, both Clojure and Scala are JVM-based, and Clojure has a very good syntax for Java interop, so all Java libraries are available to us. But, of course, Clojure community also spits out libraries…

At my company, we're actually doing that as well, but we took it further: no tiers at all. We do load-balancing at the client, every server has a public IP address, we use DNS round-robin only to distribute the login phase; after that, it's location-based with re-connect logic in the app. We don't store the entire database on every node, but we do the Cassandra-style equivalent: each datacenter has all of the data. We can store all of the data in a single datacenter. We never have to worry if the "database" is down, because the app server is also down whenever it is. We even have global visibility on all machines in a Cassandra-like design -- something that took Google's Spanner team custom GPS and atomic timekeeping hardware to achieve. We're doing it with a lightweight consensus algorithm that achieves a reliable commit timestamp globally, and NTP. We can take a global snapshot, or return a globally-consistent query across all data at any time. I think we'll see a lot of architectures built like this in the (near) future. Collapsing the tiers is great for maintainability and performance. ---- FWIW I found http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html to be...lacking. He ignores all problems related to multi-update completely. Nathan's solution does nothing to solve consistency problems except in the trivial situation where every piece of data can be "updated" (really appended) completely in isolation from any other update. You actually do need…

"Datomic is not an update-in-place system. All data is retained by default." I'm becoming more and more convinced that your canonical data store should be append-only whenever possible (see eg [1][2] for detailed arguments). It's nice to see first class support for this. [1] http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html [2] http://martinfowler.com/articles/lmax.html EDIT: Just read through the whitepaper. Looks like the indexes / storage engine form an MVCC (http://en.wikipedia.org/wiki/Multiversion_concurrency_control) key-value store, similar to Clojure's STM. Peers cache data and run datalog queries locally. This could be either an available or consistent system, depending on how cache invalidation in peers works. In the available, eventually-consistent case you have the added benefit that all queries see a consistent snapshot of the system, even if that snapshot is not totally current. Like most of Hickey's work, the whole thing seems really obvious in hindsight. It also bears a lot of similarity to Nathan Marz' recommendations for data processing and schema design.

"One technique for making software more robust is to minimize what your software depends on – the less that can go wrong, the less that will go wrong. Minimizing dependencies is more nuanced than just not depending on System X or System Y, but also includes minimizing dependencies on features of systems you are using." From http://nathanmarz.com/blog/principles-of-software-engineering-part-1.h…

1. First make it possible 2. Then make it beautiful 3. Then make it fast 4. Rinse and repeat Suffering-oriented programming (2012) — http://nathanmarz.com/blog/suffering-oriented-programming.html

Nathan worked at Twitter so while he might be wrong, I don't think it's reasonable to assume he's just naive http://nathanmarz.com/blog/leaving-twitter.html.

Nathan Marz talks about this previously, with unique insights: http://nathanmarz.com/blog/how-becoming-a-pilot-made-me-a-better-progr…

https://github.com/nathanmarz/specter http://nathanmarz.com/blog/functional-navigational-programming-in-cloj… https://en.wikibooks.org/wiki/Haskell/Lenses_and_functional_references

I heard a similar quote from Nathan Marz of Storm fame: "First make it possible, then make it beautiful, then make it fast" http://nathanmarz.com/blog/suffering-oriented-programming.html

>Despite the fact that list 'n' hashes are very common, there are times I wish I was using a thoughtful tree representation. Such a model can provide richer navigation operations. Specter[1] provides something like this for Clojure, allowing rich navigation and transformation abilities to deeply nested structures. [1]: http://nathanmarz.com/blog/functional-navigational-programming-in-cloj…

> the reality is that at small startups they simply cannot afford that kind of office space It's funny startups would feel that way yet also believe they must be located in/around San Francisco. If cost was really that big of a concern maybe they should choose to setup shop in a more affordable city (they'd also save money on salaries). But even in San Francisco the additional costs wouldn't be that much. There was a blog post[1] last year that did the math and companies save about 10% with an open office - the hit to productivity caused by an open office environment is most likely costing them more. [1] http://nathanmarz.com/blog/the-inexplicable-rise-of-open-floor-plans-i…

I really enjoyed reading about Storm too: http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned… This kind of "competition" leads to analysis paralysis though. Its much better when there is a single winner...

* Rich Hickey on Datomic, CAP and ACID [1] * Exploiting Loopholes in CAP (Michael Nygard of Relevance) [2] * How to beat the CAP theorem (by Nathan Marz of Twitter) [3] [1] https://news.ycombinator.com/item?id=5093037 [2] http://www.infoq.com/presentations/Loopholes-CAP [3] http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

Here are a few examples: http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html http://dominictarr.com/post/44516618714/working-around-the-cap-theorem… http://scale-out-blog.blogspot.ca/2012/04/disproving-cap-theorem.html Not everyone accepts the CAP theorem yet, or some have an understanding of it that is flawed enough that they think they can beat it.

Didn't Nathan Marz debunk the CAP theorem just last year? http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

> Didn't Nathan Marz debunk the CAP theorem just last year? http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html > There is another way. You can't avoid the CAP theorem, but you can isolate its complexity and prevent it from sabotaging your ability to reason about your systems. I have a different definition of "debunk".

this reminds me a lot of "How to beat the CAP theorem" http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

Seem to recall an article posted on HN some time back called "Suffering Oriented Programing."[0] I doubt this is where the idea originated but the author does a good job of relating the principal to coding. [0] http://nathanmarz.com/blog/suffering-oriented-programming.html

Kenny's initial implementation of the validation on a sample 50,000 node data graph and a specific ontology took four seconds. The goal of Kenny's project was to get a working DSL and prototype of the validation algorithm. We don't really have much performance data to compare it to either at this point. As we gather that data by using the system, we'll make it faster as necessary. (Nathan Marz's "suffering-oriented programming" post is a perfect description of this development process http://nathanmarz.com/blog/suffering-oriented-programming.html)

Nathan, I looked for the preorder button as soon as I saw your name on the landing page. For anyone who's not familiar with his work, look these over: http://www.slideshare.net/nathanmarz/storm-distributed-and-faulttolera… http://nathanmarz.com/blog/introducing-cascalog-a-clojure-based-query-… https://github.com/nathanmarz/elephantdb

Reading also this one now: "My experience as the first employee of a Y Combinator startup" http://nathanmarz.com/blog/my-experience-as-the-first-employee-of-a-y-…

In case you're wondering what cascalog is (like I was): http://nathanmarz.com/blog/introducing-cascalog/ "A Clojure-based query language for Hadoop"

DISCLAIMER: I'm one of PMC member of Apache Storm. Even though Clojure has been Nathan's first-class language, not everyone in contributors and even committers couldn't become native on Clojure. (Though I guess part of committers could feel Clojure as native.) It might not be a problem when Nathan builds most of things in Storm (I'll link his blog post which describes lesson learned from Storm on his perspective http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned… but when Storm becomes popular and also no longer his toy project, Clojure matters for scalibility of the project. Personally, I had a hard time understanding plenty of macros and even chained macros - that might not be a problem if I were allowed to have plenty of times to study Clojure and get used to it, but possible contributors don't get a chance when they're ready. In many cases, the first time they read the code on what they're using is when they encounter issues on stage/production and have to dig with stack trace. For me, I had to investigate on Storm because my team suffered from lack of documentation and also lack of knowledge on internal which ended up just taking workarounds. It was pretty hard time learning language to just read and understand the details to track down the issue. I even couldn't think of writing code on Clojure. When I decided to contribute Apache Storm, I started from connectors and CI build which wasn't written on Clojure. After some months I was able to…

One of the reasons i taught myself to program years ago, was because i wanted the power to make my life easy for myself. I am currently building a semantic supply chain for my ideas, i have being using a few notetaking apps but they do not satisfy my needs because they lack in their ability for me to query, visualize and interlink my ideas. So i was reading a paper on supply chain management and thought maybe i can implement this process for taking my ideas from inception to prototype. This is not a startup and i will use the cheapest hosting i can find, but i am thinking of allowing people to use it in the web and app form. If it gets traction, i will adjust app accordingy, if not i have a cool process to get my ideas out and visualize their process. The bottom line is: "Don't build technology unless you feel the pain of not having it." http://nathanmarz.com/blog/suffering-oriented-programming.html

1) Startups/business. If you haven't, I would read all of Paul Grahams essays. I think startups are a great option (http://www.collegeanswerz.com/risks-and-rewards-of-startups), but even if you don't end up starting one, you should understand the business. I would also highly recommend reading Rework. 2) Your site. The top part (good design is good business) and the footer (contact me) are solid, but the rest could use a lot of work. The periods in the 'Hi there' paragraph are on the next line. Could just be my browser, but something you should look into. That paragraph definitely should be "nicer". Maybe something like this: http://twitter.github.io/bootstrap/2.3.2/examples/carousel.html. The design of the case study is solid. The main problem with your site I think is the lack of information. Tell us more about yourself. What you do? What are your aspirations? Hobbies? Thoughts/opinions? Show us that you know design and programming by telling us your philosophies. Also, have a blog (http://nathanmarz.com/blog/break-into-silicon-valley-with-a-blog-1.htm… 3) Design resources. I'm new to programming and design too (~ 1 year of on and off experience), but there are some resources that have helped me a lot. - Definitely learn bootstrap (http://twitter.github.io/bootstrap/index.html). - Best resource on typography (http://practicaltypography.com/) - Best resource on color (https://www.helpscout.net/blog/psychology-of-color/) - Amazing…