Metamarkets Blog

Metamarkets | https://metamarkets.com/ | SF, NYC | On-site | Fulltime HIRING: Sr Backend Software Engineers (Scala/Java), Sr Full Stack Software Engineer, Front End Architect, DevOps, Interaction Designer, Technical Account Manager Metamarkets is the leading provider of interactive analytics for the $30B programmatic marketing industry, and inventor of the Druid open-source data store. We're a 40 person startup scaling out a next-generation streaming analytics stack to handle 1 trillion events daily, leveraging the leading open-source frameworks Kafka + Samza + Druid (developed in house) + Spark. What we offer: Comprehensive medical/dental/vision, free lunches, quarterly offsites & annual Tahoe trip, WFH Wednesdays. Apply at https://goo.gl/He15hU or mike.driscoll [at] metamarkets [dot] com.

Metamarkets - http://www.metamarkets.com/ - San Francisco, CA Looking for: Interns, Full-Time, H1B transfer-friendly Metamarkets is looking for people who are excited by data. We have a number of different engineering positions open both on the front-end and the back-end. On the front-end we're running entirely in JavaScript/CoffeeScript with Node.js for our server-side along with a framework called DVL that builds on top of D3.js. On the back-end we're using everything from Java, Scala, R, Hadoop, Pig, ZooKeeper, to Memcached. This is all to help run our own distributed, column-oriented analytical datastore called Druid. Please see http://metamarkets.com/jobs/ for more information about the jobs we have open. Questions please send to [email protected]

I believe that the note on the bottom, about machine learning, is not germane to the graph displayed. The graph is a straight visualization of hacker news post rank. The machine learning discusses this earlier blog post: http://metamarkets.com/2011/hacking-hacker-news-headlines/ I don't understand why the authors combined boosting with stochastic-gradient descent. Boosting means you do greedy, forward-selection of features. You add one feature at a time, the one that has the largest gradient with respect to the loss. i.e. you pick the feature that looks like it will fit the data the best, given what the model currently knows, and then add the feature to the model. You can then choose the optimal weight for the feature using a line search. Why is SGD necessary in this scenario? After you picked your features, did you go back and re-adjust the weights? I'm not sure why this is necessary. If you are going to do SGD over many features with strong normalization, why not do it over the entire feature vector and see which features disappear?

- From the event page - Tech Talk: Eric Tschetter on Druid - Distributed Exploration of High Dimensional Data Druid is a distributed system in use at Metamarkets (http://www.metamarkets.com) to facilitate rapid exploration of high dimensional data. They use Druid to expose impression monetization data to internet publishers along any arbitrary combination of demographic, content and sales-based dimensions. One Druid cluster currently exposes a data set of >15 billion rows of data representing >500 billion impressions in hypercubes of varying dimensionality (largest is 28 dimensions) while allowing for exploration using top lists and timeseries in sub-second latencies. The tech talk will be a discussion of the design considerations and architecture of the system. http://metamarkets.com/2011/druid-part-i-real-time-analytics-at-a-bill… About Eric Tschetter Eric Tschetter is the lead architect of Druid, Metamarkets' distributed, in-memory database. He held senior engineering positions at Ning and LinkedIn before joining Metamarkets. At LinkedIn, Eric productized LinkedIn's PYMK with Hadoop. He holds bachelors degrees in Computer Science and Japanese from the University of Texas at Austin, and a M.S. from the University of Tokyo in Computer Science.

Here are a few production users: https://cwiki.apache.org/confluence/display/SAMZA/Powered+By The Metamarkets team wrote a nice post on their use of Samza a few days ago: https://metamarkets.com/2015/simplicity-stability-and-transparency-how…

The way business logic in the system is currently designed , there are no 'common ones', except for some ids (already indexed) used in other process rather than this dynamic filter. Based on that we also evaluated the approach of using something else like druid [1] [2] that is built for reading performance, but I am still studying possibilities and have no idea about the impact and problems a change like that would impose. [1] http://druid.io/ [2] https://metamarkets.com/2014/building-a-data-pipeline-that-handles-bil…

Obligatory repost - "Hacking Hacker News Headlines" from May 2011, examining the significance of language in story headlines: http://metamarkets.com/2011/hacking-hacker-news-headlines/

It's used by Metamarkets in production (http://metamarkets.com/); I've seen talks about it at Strata and they're getting really impressive query times on terascale problems. Shameless plug: if you're interested in fast in-memory analytics and more of the PyData bent, I'm building a system with some similarities to Druid at my company (http://www.datapad.io/) but more focused on optimizing single-node performance (i.e. minimizing EC2 expenses) on medium data (single columns of tables typically fit in memory) than high scalability / real time ingest.

http://d3js.org/ => http://metamarkets.com/2011/hacking-hacker-news-headlines/ => https://news.ycombinator.com

Metamarkets also open sourced their data store today. High performance, distributed query database. http://metamarkets.com/category/technology/druid/

Here's a list of relevant links I've compiled over the past year or two: - http://nathanael.hevenet.com/the-best-time-to-post-on-hacker-news-a-co… - http://minimaxir.com/2014/02/hacking-hacker-news/ - http://minimaxir.com/2014/10/hn-comments-about-comments/ - http://karpathy.ca/myblog/2013/11/27/quantifying-hacker-news-with-50-d… https://cs.stanford.edu/people/karpathy/hn_analysis.html - http://metamarkets.com/2011/hacking-hacker-news-headlines/ - http://danluu.com/randomize-hn/ - http://www.bayesianwitch.com/blog/2013/why_hn_shouldnt_use_randomized_… - http://camdp.com/blogs/multi-armed-bandits - http://gkosev.blogspot.com/2012/08/fixing-hacker-news-mathematical-app… - http://blog.rjmetrics.com/2012/10/17/surprising-hacker-news-data-analy… - http://blog.rjmetrics.com/2012/10/24/how-to-get-on-the-front-page-of-h… - http://blog.datadive.net/which-topics-get-the-upvote-on-hacker-news/ - https://news.ycombinator.com/item?id=8790134 - https://porter.io/blog/hackernews-cheaters-catch-me-if-you-can/ https://news.ycombinator.com/item?id=9332889 - http://www.righto.com/2013/11/how-hacker-news-ranking-really-works.htm… - https://news.ycombinator.com/item?id=9336759 https://news.ycombinator.com/item?id=9333611 - http://www.almostinfinite.com/other/hacker-news-new-page-scroll-of-dea…

I know no solution doing all we need, only some interesting components. Here are some more links: storing data * couchdb http://couchdb.apache.org/. Good if you already know the analysis you want to do. drawback is that indexes you create (= materialized views) are monodimensional. a multidimensional indexing on top of it would simply be great. Not great if you need real real time reporting and your data change a lot. * mongodb: more flexible than couchdb for ad-hoc querying * graph databases: neo4j, orientdb, ... might be a good alternative to OLAP * ElasticSearch http://www.elasticsearch.org/ lets you have faceted indexes (= index and retrieve facts according to dimensions values). Pluggable to a many storage solutions reporting / visualizations * d3.js http://mbostock.github.com/d3/ far above other charting libraries * Simile Exhibit http://simile-widgets.org/exhibit3/ Great for faceted browsing (facet = dimension) * Cube http://square.github.com/cube/ Time series visualizations * mind gapper https://github.com/RandomEtc/mind-gapper-js visualizations la Hans Rosling's Gapminder * dejavis http://beta.dejavis.org/ looks offline for the moment Big data * Hadoop + data-warehousing / Querying solutions Apache Pig and/or hive ETL (nothing exceptional here) * Yahoo Pipes * Python flow * Google Refine Saas * Bime http://bimeanalytics.com/ * Druid http://metamarkets.com/2011/druid-part-i-real-time-analytics-at-a-bill…