N
Nextdoor Engineering Blog
engblog.nextdoor.com · Company blog · on HN since 2014
Nextdoor's engineering team writes about its infrastructure, including task workers, continuous deployment, mobile releases, and tooling.
What HN saysCommenters cite its posts on dropping cron jobs and replacing Celery with a custom taskworker when discussing job scheduling and why companies build their own.
23HN people shared it
6linked it in comments
34stories submitted
441story points
2 years agolast shared
2014people per year2026
Most-shared posts
- 1We don’t run cron jobs at Nextdoor3 people · 5 points · first shared 2015 · 1 comments on HN
- 2Nextdoor taskworker simple efficient amp scalable ac4f7886957b3 people · first shared 2018
- 3We Don’t Run Cron Jobs (2016)2 people · 177 points · first shared 2018 · 140 comments on HN
- 43 Hard Lessons from Scaling Continuous Deployment to a Monolith2 people · 8 points · first shared 2017 · discussion on HN
- 5Managing Mobile App Releases2 people · 3 points · first shared 2015 · discussion on HN
- 6A powerful way to integrate Hubot with Slack2 people · 2 points · first shared 2015 · discussion on HN
- 7Show HN: Bender – Let's Standardize Serverless ETL1 people · 52 points · first shared 2017 · 21 comments on HN
- 8The technology behind Nextdoor1 people · 46 points · first shared 2015 · 10 comments on HN
- 9Maps on Android1 people · 33 points · first shared 2015 · 11 comments on HN
- 10Building a Scalable Geospatial Infrastructure1 people · 15 points · first shared 2015 · discussion on HN
- 11What to log in production environments1 people · 13 points · first shared 2015 · 1 comments on HN
- 12Structured Logs at Nextdoor1 people · 10 points · first shared 2015 · discussion on HN
- 13Easy A/B testing at Nextdoor1 people · 10 points · first shared 2015 · 1 comments on HN
- 14How Nextdoor made a 10x improvement in release times with Docker and Amazon ECS1 people · 9 points · first shared 2017 · discussion on HN
- 15Data Migrations Don’t Have to Come with Downtime1 people · 9 points · first shared 2020 · discussion on HN
What HN people wrote when they linked it
solumos · 2020-11-11
It seems that Nextdoor also had issues with celery[0].
"Scalability" is a great scapegoat for making dubious decisions, but my guess here would be the "task priority" requirement.
[0] https://engblog.nextdoor.com/nextdoor-taskworker-simple-efficient-amp-…
izyda · 2015-08-12
Looks good. Very much looking forward to this being production ready. I am currently looking for a scheduler that provides a better UI/UX than cron alone.
Currently, what do you all use for this sort of thing?
Nextdoor has a not-yet-open-sourced solution: https://engblog.nextdoor.com/2015/06/10/we-do-not-run-cron-jobs-at-nex… that looks great.
Does anyone have any experience with Yelp's tron? https://github.com/Yelp/Tron
deizel · 2018-08-22
Ditto, especially in combination with Python/Django, as used by Nextdoor. Ironically, they had already removed Celery from their stack a few years prior. https://engblog.nextdoor.com/nextdoor-taskworker-simple-efficient-amp-…
lhr0909 · 2018-08-21
I think they have a task worker system built back in 2014[1], so they need to have something custom working with it as well. Back then I think they really didn't have many options, but if they were to do it again now, I think either AWS Lambda or AWS Batch will serve this type of scheduled job cases very well.
[1]: https://engblog.nextdoor.com/nextdoor-taskworker-simple-efficient-amp-…
slap_shot · 2019-10-10
I don't really get into this argument anymore - I'm convinced opinions like this are held tighter than religion or politics.
Here are a few thoughts, written by others, that I would probably bring up if I wanted to get into this argument:
https://medium.com/eshares-blog/why-you-shouldnt-use-cron-jobs-5d76001…
https://engblog.nextdoor.com/we-don-t-run-cron-jobs-at-nextdoor-6f7f9c…
https://medium.com/videoamp/what-we-learned-migrating-off-cron-to-airf…
https://medium.com/@rbahaguejr/airflow-a-beautiful-cron-alternative-or…
As a point of reference, I run a company that brokers terabytes of mission critical data each month for our customers. I've advised dozens of companies on how to properly ensure scheduled jobs are maintainable, instrumented, logged, and reliable. The defenses I've heard for "professional" software and systems engineers using cron in production software are appalling and I find the behavior negligent and to be a terminable offense.
folz · 2019-10-01
Nextdoor | San Francisco, CA; Toronto, ON, Canada | Full-time, Onsite | https://nextdoor.com/
Nextdoor is the world’s largest social network for the neighborhood, serving 247,000 neighborhoods across eleven countries. We recently closed a $170MM series F where Mary Meeker joined our board, and we officially launched in Canada. To support that, we're opening our first international engineering team in Toronto!
Here's a look at our current stack: https://engblog.nextdoor.com/what-is-the-technology-behind-nextdoor-in…
Some of our open positions in SF:
* Software Engineer: https://about.nextdoor.com/careers-list/?gh_jid=424&gh_src=f342d7f…
* Senior Software Engineer: https://about.nextdoor.com/careers-list/?gh_jid=1228742&gh_src=f34…
* Data Scientist: https://about.nextdoor.com/careers-list/?gh_jid=1509433&gh_src=f34…
* Machine Learning Engineer: https://about.nextdoor.com/careers-list/?gh_jid=1241186&gh_src=f34…
* Engineering Manager: https://about.nextdoor.com/careers-list/?gh_jid=1215063&gh_src=f34…
* Product Manager, Community Vitality: https://about.nextdoor.com/careers-list/?gh_jid=1621033&gh_src=f34…
And in Toronto:
* Software Engineer, Android: https://about.nextdoor.com/careers-list/?gh_jid=1851532&gh_src=f34…
* Software Engineer, iOS: https://about.nextdoor.com/careers-list/?gh_jid=1851506&gh_src=f34…
* Software Engineer: https://about.nextdoor.com/careers-list/?gh_jid=1851545&gh_src=f34…
We're…