AdRoll Tech Blog
tech.adroll.com · Company blog · on HN since 2013
AdRoll's engineering team writes on data pipelines, containers, probabilistic data structures, Erlang/Elixir and front-end infrastructure at an ad-tech company.
What HN saysHN links it for data pipeline architecture and HyperLogLog posts; one commenter cautions that HLL intersections give large errors, and it appears in hiring threads.
26HN people shared it
10linked it in comments
39stories submitted
621story points
7 years agolast shared
2013people per year2026
Most-shared posts
- 1How to Run a Front-End Infrastructure Team5 people · 10 points · first shared 2017 · discussion on HN
- 2D is for Data Science4 people · 30 points · first shared 2014 · 1 comments on HN
- 3HyperLogLog and MinHash3 people · 47 points · first shared 2014 · 7 comments on HN
- 4Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot Instances2 people · 51 points · first shared 2015 · 5 comments on HN
- 5Factorization Machines2 people · 30 points · first shared 2015 · 14 comments on HN
- 6Querying Data in Amazon S3 Directly with User-Space Page Fault Handling2 people · 15 points · first shared 2016 · 1 comments on HN
- 7Quaff that potion: saving $millions with Elixir and Erlang2 people · 12 points · first shared 2018 · discussion on HN
- 8Streaming Petabytes of Data in Realtime with Kinesis2 people · 10 points · first shared 2015 · discussion on HN
- 9Managing Containerized Data Pipeline Dependencies with Luigi2 people · 8 points · first shared 2015 · discussion on HN
- 10Running large batch processing pipelines on AWS Batch2 people · 3 points · first shared 2018 · discussion on HN
- 11Collider: A New Frontier in Testing2 people · 1 points · first shared 2016 · discussion on HN
- 12Rollup major learnings2 people · first shared 2016
- 13Blog2 people · first shared 2017
- 14TrailDB – An Efficient Library for Storing and Processing Event Data1 people · 212 points · first shared 2016 · 54 comments on HN
- 15Hologram: taking EC2 Instance Roles everywhere1 people · 84 points · first shared 2014 · 26 comments on HN
What HN people wrote when they linked it
goo · 2018-02-01
Looks like that link got cut off, likely copied from a prior post. The following link should resolve: http://tech.adroll.com/blog/frontend/2017/08/29/how-to-run-a-front-end…
vtuulos · 2016-12-01
We have been using this architecture for containerized batch processing for more than a year:
http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html
It is very convenient to be able to define any jobs as Docker containers, using the best language for the task, and define dependencies between their inputs and output explicitly, not being tied to any particular paradigm like MapReduce.
I am more than happy to let AWS handle plumbing for this instead of having to maintain and operate our custom solution.
t1m · 2015-05-04
Note that intersections using HLLs don't work in general. The error rates become gigantic for sets that have small intersections, or whose cardinality differs greatly.
A general solution to cardinality estimations for intersections is to just use HLLs for union operations, and keep a MinHash sketch for each key as well to perform intersections.
There is a very good analysis of this technique over at http://tech.adroll.com/blog/data/2013/07/10/hll-minhash.html
aaaron · 2014-03-01
AdRoll - San Francisco, CA. H1B welcome.
http://tech.adroll.com
This week named one of the fastest growing technologies online by BuiltWith (https://blog.builtwith.com/), we help over 10,000 companies collect, analyze and act on their customer data. Everyone from New Relic to Rackspace to Twilio uses AdRoll.
With worldwide scale and over 20B events to process a day, ad tech has some of the most interesting problems you'll find anywhere. AdRoll's engineers, data scientists and product people are conquering problems in big data (HBase), machine learning, real-time processing (Erlang/Storm/Kafka) and UX (Backbone).
Email us at [email protected] or see our tech blog http://tech.adroll.com
dialtone · 2018-02-01
AdRoll | San Francisco | On-site/remote | Full-time
If you like developing open-source code, languages such as Python, Go, JS, C, D, Lua, Erlang, AWS, petabytes of data, and distributed low-latency systems, this may be your dream job.
This time we are particularly interested in finding full stack web developers with good JavaScript experience and experienced developers, tech leads and data scientists with great math knowledge and coding skills. This is a really unique opportunity to get to work with a massive scale (thousands of instances on AWS), low latency (real-time bidding with 100ms max latency and 70B requests daily, real-time machine learning with 1ms max latency), mission-critical systems (this is how we make money) and enjoy working on a strong frontend development team (http://tech.adroll.com/blog/frontend/2017/08/29/how-to-run-a...).
Learn more about us here http://tech.adroll.com/blog/
I am happy to tell you more over coffee in SF or by email, [email protected]
Timothee · 2014-11-10
I like the idea behind this because I am also very sensitive to that type of things. Probably over-sensitive but I just worry that if you let one thing go, it becomes a slippery slope and you then have a mess of a repo.
That being said, if you care about these things, I wonder if these checks are best left to a pre-commit hook. It removes noise from the commit history and the PRs and forces people to think about it right away, rather than being corrected after the fact.
I guess having that done on a server has the benefit of not having to worry about keeping the linters' version up-to-date/homogenous over all the devs' machines.
At work, we have JSCS (https://github.com/jscs-dev/node-jscs) and SCSS-lint (https://github.com/causes/scss-lint) as part of our pre-commit hook (on top of editor plugins) and that has been great honestly. It decreased the PR noise a lot and I feel it has been good for new hires since it avoids having the first PR comments being about style issues.
I wrote a post about how I added JSCS in the pre-commit hook by the way: http://tech.adroll.com/blog/web/2014/03/05/adding-jscs-to-your-commit-… It should be easily extendable to other linters.
ZenoArrow · 2018-02-04
For some languages, that distinction doesn't seem to apply. For example, it seems that the D language can be treated as both a scripting language and a system language, as the speed of compilation + execution can be comparable to that of scripting languages. For evidence of this, take a look at the first D versus Python comparison in this article, note that both scripts have comparable execution times (both finished running in under a second):
http://tech.adroll.com/blog/data/2014/11/17/d-is-for-data-science.html
It should be noted that D has a number of different compilers, offering the user the choice between compilation speed and optimisation.
nemothekid · 2018-01-08
HLL intersections using MinSketch has been around for a while now, I remember writing a Go implementation a couple years back based on a AdRoll blog post (http://tech.adroll.com/blog/data/2013/07/10/hll-minhash.html). They also open sourced their library (https://github.com/AdRoll/cantor) as well.
Generally in the big data space, libraries like this usually come out for Java first.
giancarlostoro · 2017-05-25
Based on that I would look at Funatics since they seem to have the "bigger" traffic, or even AdRoll...
http://tech.adroll.com/blog/data/2014/11/17/d-is-for-data-science.html
Edit:
To be fair a lot of the companies there look like they handle heavier loads. Also Garbage Collection is optional, and there are alternatives to the "standard library" for D that others have made that are probably usable without GC. Some people have done successful embedded systems programming without the GC, I remember one guy talking about it on the D irc channel.
juriga · 2016-09-01
AdRoll | San Francisco, CA | Full time
Senior UI Engineer (React, Redux, JavaScript ES6/ES7, private npm, Browserify/Webpack, SASS)
Our frontend team is looking for a seasoned engineer with a passion for world-class UX design and elegant UI development workflows. As part of this small team of senior developers and designers, you get to make a big impact building reusable React components and implementing our new style guide across all our products.
AdRoll is one of the most widely used adtech companies, with 25k+ companies using the product globally and $100M+/yr in revenue.
As we build, we're also sharing our code...
http://tech.adroll.com/blog/frontend/2015/12/21/gulp-react-docs.html
...and learnings with the open source community:
http://tech.adroll.com/blog/frontend/2015/11/19/rollup-major-learnings…
I'm happy to share more details via email ([email protected]), Twitter (@jtuulos) or over coffee in SF.