Lyft Engineering

Yesterday a friend just sent me a great post on tuning gevent with gunicorn: https://eng.lyft.com/gevent-part-3-performance-e64303fa102b gevent workers are best suited when your app spends a lot of time waiting on I/O (as you note), and they recommend setting `--worker-connections` to the ratio of {network IO time}/ {CPU time}. If you set that too high, then it can result in higher latency for requests.

Lift Engineering blog - not too technical but interesting b/c we all enjoy the product: https://eng.lyft.com/ start here: https://eng.lyft.com/matchmaking-in-lyft-line-9c2635fe62c4

I definitely become more attracted to companies with high quality blog posts on things I'm interested in. The right blog is a really big signal that I'd be interested in the work at the company. Besides picking the next company, great blog posts can be a real help in sanity checking the work that's happening in our team. I've noticed even senior engineers treating good blog posts from big companies as "See, we're not crazy". Right now I'm operating in the data science platform space at my company, and have been reading a lot more company blogs than usual. Wayfair is a company I now know of after an engineer I really respect shared on Twitter "Bayesian Product Ranking at Wayfair"[1]. Lyft, on the other hand, disappointed me with "How Lyft Designs the Machine Learning Software Engineering Interview"[2]. That post sounds like it could have been victim to what Dan notes as: > Approval/editing process mainly de-risks posts, removes references to specifics, makes posts vaguer and less interesting to engineers 1. https://tech.wayfair.com/data-science/2020/01/bayesian-product-ranking… 2. https://eng.lyft.com/how-lyft-designs-the-machine-learning-software-en…

Lyft is hiring for all positions, including mobile, frontend, backend, infrastructure, and product roles. We're working on interesting and challenging engineering and product problems to improve people's lives with great transportation. I'm specifically looking for several information security roles: • An application security engineer (https://www.lyft.com/jobs/software-engineer-application-security) who’s excited about helping Lyft ship safe software quickly. You’ll help product and engineering teams at Lyft develop new features and products that are innovative and protect our users, build systems and tools to make it easy for engineers at Lyft to develop safe software by default, and promote security throughout the company. • A detection & response engineer (https://www.lyft.com/jobs/software-engineer-security-detection-respons… who’s excited about protecting Lyft and our users through swift detection of and response to security incidents. You’ll develop and automate techniques to hunt for and alert on attacker behavior with high signal-noise ratio, test our detection and response capabilities to improve our tools and processes, and act as a calm and rational incident commander when incidents do occur. • A data privacy engineer (https://www.lyft.com/jobs/software-engineer-data-privacy) who’s excited about protecting our users by ensuring the data we collect is used appropriately without inadvertent or malicious misuse or disclosure. You’ll develop tools to enable…

It also doesn't show effective accesses from users/groups/roles/things-across-accounts; I've found that even Access Analyzer gets it very wrong sometimes. I wrote a bit on this here: https://eng.lyft.com/iam-whatever-you-say-iam-febce59d1e3b.

Part 2: https://eng.lyft.com/causal-forecasting-at-lyft-part-2-418f1febca5a

Thanks for the feedback! One of the creators here :) Our initial thought is to create a solution similar to what Lyft has done (https://eng.lyft.com/scaling-productivity-on-microservices-at-lyft-par… to enable using staging as a dev environment and sharing it with many developers at once. We do want people to use at us production later on, but right now we know there might be compliance/security issues.

I'm losing count of all the metadata management systems being open sourced. First there was Wherehows out of LinkedIn [1]. Then there was Amundsen out of Lyft, which is now stemma.io [2]. Then there was Marquez out of WeWork [3]. Now we have DataHub out of LI. It's a little crazy how underserved this problem was several years ago and how over served it's becoming. I use Amundsen and am pretty happy with it. [1] https://engineering.linkedin.com/blog/2016/03/open-sourcing-wherehows-… [2] https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d… [3] https://marquezproject.github.io/marquez/

Yes many of the top tech companies are now hiring across Europe (and in fact have been). Ex. https://eng.lyft.com/introducing-lyft-engineering-kyiv-2884d1bba69a Salaries are definitely adjusted. To get a feel for salaries: - https://www.levels.fyi/Salaries/Software-Engineer/Netherlands/ - https://www.levels.fyi/Salaries/Software-Engineer/Ukraine/ - https://www.levels.fyi/Salaries/Software-Engineer/London/

Here are a few software companies moving to next.js. https://nextjs.org/case-studies/hulu https://nextjs.org/showcase/uber https://eng.lyft.com/changing-lanes-how-lyft-is-migrating-100-frontend…

Link to the article: https://eng.lyft.com/changing-lanes-how-lyft-is-migrating-100-frontend…

Lyfts announcement of the envoy proxy: https://eng.lyft.com/announcing-envoy-c-l7-proxy-and-communication-bus… Interesting read. I actually wrote a consul SDS integration for envoy a couple of years back at a previous employment. Hasihicorp simple local setup: https://learn.hashicorp.com/consul/developer-mesh/connect-envoy

Have Lyft migrated all their workflows from Airflow to Flyte? Or does Airflow still play a role alongside Flyte? Was assuming Lyft is running workflows in Airflow from this post https://eng.lyft.com/running-apache-airflow-at-lyft-6e53bb8fccff

Maybe Lyft varies it's interviews, but: 1) https://whiteboardfree.com/job_posts/52 - Lyft - Data Engineering 2) https://eng.lyft.com/interviewing-tips-for-android-engineers-f01ce7fba… - "Expect a few coding questions either on a whiteboard " ( Dec 21, 2017 ) Pretty sure Netflix (also has a job listed) does whiteboard based evaluation too.

Author here, ask me any questions you might have. We used this internally to convert a few repos. It's not perfect but gives you a head start for converting to TS. I also wrote about our experience with TypeScript at Lyft if you're interested: https://eng.lyft.com/typescript-at-lyft-64f0702346ea

The author's ideal language seems to also describe TypeScript https://eng.lyft.com/typescript-at-lyft-64f0702346ea

It would appear that Lyft is making some strong ties with Google as of late. Between this and the announcement of the Lyft partnership with IBM and Google to work on some sort of Kubernetes router mesh [1] I am starting to see a trend. "We are excited to announce that we are working in partnership with both Google and IBM to bring Envoy to Kubernetes. Fun fact: there are now more people working on Envoy at Google than there are at Lyft! We have a lot of other things planned with Google that we will be able to share more about in the coming months." [1] https://eng.lyft.com/envoy-7-months-later-41986c2fd443

Blog announcement: https://eng.lyft.com/announcing-ratelimit-c2e8f3182555

> No way on earth a user app should be getting its own time This practice is becoming increasingly common for "time-sensitive applications": https://eng.lyft.com/freezing-time-6ebe8ffe3321

Related: https://eng.lyft.com/automated-style-review-with-linty-fresh-17c6cd9d2…

Lots of cool projects around secrets lately: * Vault (by Hashicorp) [go] https://hashicorp.com/blog/vault.html * Confidant (by Lyft) [Python] https://eng.lyft.com/announcing-confidant-an-open-source-secret-manage… * Keywhiz (by Sqaure) [Java] https://square.github.io/keywhiz/

We only use KubernetesOperators, but this has many downsides, and it's very clearly a 2nd thought of the Airflow project. It creates confusion because users of Airflow expect features A, B, and C, and when using KubernetesOperators they aren't functional because your biz logic is separated. Eg., if your biz logic knows what S3 it talks to in an external task, how can Airflow? So now its Dataset feature is useless. There are a number of blog posts echoing a similar critique[1]. Using KubernetesOperators creates a lot of wrong abstractions, impedes testability, and makes Airflow as a whole a pretty overkill system just to monitor external tasks. At that point, you should have just had your orchestration in client code to begin with, and many other frameworks made this correct division between client and server. That would also make it easier to support multiple languages. According to their README: https://github.com/apache/airflow#approach-to-dependencies-of-airflow > Airflow has a lot of dependencies - direct and transitive > The important dependencies are: SQLAlchemy, Alembic, Flask, werkzeug, celery, kubernetes Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server? [1] Anecdotes from various posts - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-a… - https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-comparing-fl… -…