Teads Engineering
medium.com/teads-engineering · Company blog · on HN since 2017
Teads' engineers write about cloud costs, Kafka and BigQuery analytics, Spark tuning, and scaling infrastructure.
What HN saysCommenters link it for AWS cost optimization advice and Spark performance tuning tips.
16HN people shared it
3linked it in comments
30stories submitted
267story points
3 years agolast shared
2017people per year2026
Most-shared posts
- 1Generating UUIDs at Scale on the Web3 people · 15 points · first shared 2020 · 1 comments on HN
- 2Real-life AWS infrastructure cost optimization strategy3 people · 14 points · first shared 2017 · discussion on HN
- 3Scaling Graphite in a Cloud Environment3 people · 9 points · first shared 2019 · discussion on HN
- 4Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery2 people · 179 points · first shared 2018 · 68 comments on HN
- 5The most accurate way to schedule a function in a web browser2 people · 8 points · first shared 2020 · discussion on HN
- 6Evaluating the carbon footprint of a software platform hosted in the cloud2 people · 3 points · first shared 2020 · discussion on HN
- 7Easy Cassandra Scaling with Terraform, Chef and Rundeck2 people · 2 points · first shared 2018 · discussion on HN
- 8Apache Spark performance tuning from the trenches (v2.2.0)2 people · 1 points · first shared 2018 · discussion on HN
- 9Machine Learning Use Cases from the AdTech Industry Using Spark1 people · 15 points · first shared 2017 · discussion on HN
- 10When Spark's MLlib Isn't Enough – Machine Learning at Teads1 people · 4 points · first shared 2017 · discussion on HN
- 11Serve Analytics data with Redshift, a counterintuitive choice1 people · 3 points · first shared 2019 · discussion on HN
- 12Scaling your on-duty team1 people · 2 points · first shared 2018 · discussion on HN
- 13Updating to Spark 3.0 in production: Breaking changes and expected improvement1 people · 2 points · first shared 2020 · discussion on HN
- 14Building an AWS EC2 Carbon Emissions Dataset1 people · 2 points · first shared 2021 · discussion on HN
- 15Lessons learned while optimizing Apache Spark aggregation jobs1 people · 2 points · first shared 2019 · discussion on HN
What HN people wrote when they linked it
A21z · 2020-07-11
Blog article about real-life AWS cost optimization strategy :
https://medium.com/teads-engineering/real-life-aws-cost-optimization-s…
TL;DR:
- Monitor your daily costs
- Cloudify your workloads
- Use reservations, Spot instances, saving plans
- Look around for unused infrastructure
- Optimize S3 consumption
maslam · 2018-12-24
Karthik, I'm no Spark expert but almost all advice I read is to avoid UDFs if at all possible. Examples below:
- https://medium.com/teads-engineering/spark-performance-tuning-from-the…
- https://www.inovex.de/blog/efficient-udafs-with-pyspark/
Lazare · 2021-05-04
There's also the risk of bad randomness sources and/or bugs.
One popular UUID library got a bug report stating: "We are generating about 1M UUID4 a day, and we are getting several hundred collisions a day". And so they were; turned out to be a bug/weird interaction between the OpenSSL library they were using for randomness and forking. (Details here, although it was all fixed years ago of course: https://github.com/ramsey/uuid/issues/80)
On paper, you should never, ever, ever see a collision when generating a mere million v4 UUIDs a day, much less hundreds of collisions. But that doesn't mean it can't happen!
This is also an interesting bit of analysis; comes from a company that processed a lot of UUIDs generated in browsers, checked, and discovered about 5 collisions per million UUIDs. Again, not what you'd naively expect! (Turned out to be mostly driven by misbehaving crawlers.) https://medium.com/teads-engineering/generating-uuids-at-scale-on-the-…