Vimeo Engineering
medium.com/vimeo-engineering-blog · Company blog · on HN since 2016
Vimeo's engineering team writes about video streaming, codecs like AV1, load balancing, PHP, and infrastructure work at the company.
What HN saysHN commenters link it for Vimeo's explanation of why plain consistent hashing gave uneven load, plus posts on in-process caching and AV1.
24HN people shared it
4linked it in comments
26stories submitted
92story points
3 years agolast shared
2016people per year2026
Most-shared posts
- 1Improving load balancing with a new consistent-hashing algorithm6 people · 12 points · first shared 2016 · discussion on HN
- 2Vimeo Engineering: It’s not legacy code – it’s PHP5 people · 39 points · first shared 2020 · 2 comments on HN
- 3Behind the Scenes of AV1 at Vimeo3 people · 4 points · first shared 2019 · discussion on HN
- 4How Vimeo turned a metric for video quality into better adaptive streaming2 people · 4 points · first shared 2022 · discussion on HN
- 5ClickHouse Is in the House2 people · 3 points · first shared 2023 · discussion on HN
- 6Uncovering PHP Bugs with Template – Vimeo Engineering Blog2 people · 2 points · first shared 2019 · discussion on HN
- 7Vimeo Joins the Alliance for Open Media1 people · 15 points · first shared 2018 · discussion on HN
- 8Upgrading images on Vimeo1 people · 4 points · first shared 2021 · discussion on HN
- 9Scalable codec testing with Are We Compressed Yet?1 people · 2 points · first shared 2020 · discussion on HN
- 10Android Instant Apps, step-by-step: how Vimeo went about it1 people · 2 points · first shared 2017 · discussion on HN
- 11Vimeo Engineering: Faster application tests with MySQL simulation1 people · 1 points · first shared 2021 · discussion on HN
- 12Vimeo is adopting tus1 people · 1 points · first shared 2017 · discussion on HN
- 13Fixing code that ain’t broken (2018)1 people · 1 points · first shared 2019 · discussion on HN
- 14A deep dive into Vimeo’s storage strategy for videos1 people · 1 points · first shared 2023 · discussion on HN
- 15Picture-perfect thumbnails for all videos1 people · 1 points · first shared 2021 · discussion on HN
What HN people wrote when they linked it
zubspace · 2017-06-27
I'm not an expert in this field, but an engineer of vimeo went into detail, why this approach did not work for them. [1]
Problem with consistent hashing:
However, consistent hashing comes with its own problem: uneven distribution of requests.
Because of its mathematical properties, consistent hashing only balances loads about as
well as choosing a random server for each request, when the distribution of requests is
equal. But if some content is much more popular than others (as usual for the internet),
it can be worse than that.
Problem with Power of 2 Load Balancing:
Why wasn’t there a way to say “use consistent hashing, but please don’t overload any
servers”? As early as August 2015, I had tried to come up with an algorithm based on
the power of two random choices that would do just that, but a bit of simulation said
that it didn’t work. Too many requests were sent to non-ideal servers to be worthwhile.
Instead, he used something called Consistent Hashing with Bounded Loads.
[1] https://medium.com/vimeo-engineering-blog/improving-load-balancing-wit…
ddorian43 · 2020-08-10
Distributed in-process cache can make things very fast. See vimeo with group cache: https://medium.com/vimeo-engineering-blog/video-metadata-caching-at-vi…
Or in-process distributed rate limiting: https://github.com/mailgun/gubernator
https://github.com/golang/groupcache:
groupcache is in production use by dl.google.com (its original user), parts of Blogger, parts of Google Code, parts of Google Fiber, parts of Google production monitoring systems, etc.
There was also a paper by google that I can't find right now.
Edit: Found it https://blog.acolyer.org/2019/06/24/fast-key-value-stores/
mindfreeze · 2020-06-13
muglug · 2019-08-12
> until you get arrays which can take anything
There's a widely-used PHPDoc standard for annotating arrays: `Foo[]`, and there are at least three PHP static analysis tools that can correctly interpret those annotations: Psalm (https://psalm.dev), Phan (https://github.com/phan/phan) and PHPStan (https://github.com/phpstan/phpstan).
Phan and Psalm also support a docblock syntax for generics - I wrote about it here: https://medium.com/vimeo-engineering-blog/uncovering-php-bugs-with-tem…