W
WePay Engineering
wecode.wepay.com · Company blog · on HN since 2016
WePay Engineering is the company engineering blog of a payments firm, covering data pipelines, databases, Kafka, Airflow, and distributed systems.
What HN saysHN links it for posts like Waltz, its distributed write-ahead log, and for WePay job listings, with commenters citing it on scaling problems.
16HN people shared it
5linked it in comments
22stories submitted
505story points
2 years agolast shared
2016people per year2026
Most-shared posts
- 1Waltz: A Distributed Write-Ahead Log3 people · 173 points · first shared 2019 · 44 comments on HN
- 2Streaming Databases in Real-Time with Kafka, Debezium, and MySQL2 people · 113 points · first shared 2017 · 30 comments on HN
- 3Balancing the Books at Scale2 people · 5 points · first shared 2020 · discussion on HN
- 4Improving Airflow UI Security2 people · 3 points · first shared 2018 · 1 comments on HN
- 5Streaming Cassandra at WePay1 people · 113 points · first shared 2019 · 43 comments on HN
- 6Highly Available MySQL Clusters at WePay1 people · 68 points · first shared 2019 · 6 comments on HN
- 7Building WePay's data warehouse using BigQuery and Airflow1 people · 5 points · first shared 2016 · discussion on HN
- 8Software Engineering Interviewing at WePay1 people · 4 points · first shared 2016 · discussion on HN
- 9Training machine learning models with Airflow and BigQuery1 people · 4 points · first shared 2016 · discussion on HN
- 10How WePay Uses Elixir for CLI1 people · 3 points · first shared 2018 · discussion on HN
- 11Using Google Cloud BigQuery at WePay1 people · 3 points · first shared 2016 · discussion on HN
- 12Loading Data from Kafka into BigQuery with Kafka Connect1 people · 2 points · first shared 2016 · discussion on HN
- 13Building WePay's Webhook Delivery System with Google Cloud Pub/Sub1 people · 2 points · first shared 2016 · discussion on HN
- 14Airflow at WePay1 people · 2 points · first shared 2016 · discussion on HN
- 15Effective Software Design Documents1 people · 2 points · first shared 2020 · discussion on HN
What HN people wrote when they linked it
jjmiller · 2017-05-01
WePay | YCS09 | Redwood City, CA | Senior Software Engineers, Site Reliability Engineers | ONSITE
We are looking for Sr. Software Engineers & SRE's to work on a handful of from scratch, greenfield development projects -- developing new micro-services on top of our new SOA to working on building our data infrastructure from the ground up (Google Cloud Platform using Kafka, Airflow and BigQuery)...
We are looking strong problem solvers to join our rapidly growing team!
Curious to know what we’re working on here at WePay? Check out our engineering blog
https://wecode.wepay.com/
Any questions, email miller [at] wepay [dot] com - OR - Apply at https://go.wepay.com/careers
duncan_bayne · 2020-07-15
Sure, if the provider of the gRPC service wishes to also offer an HTTP gateway, that's true.
But it's trivially true in the case where the provider does so wish (as it would be true of any protocol with a fully-functional HTTP gateway infront of it), and it's untrue in the case where the provider doesn't.
And the provider has to do extra work:
"[offering an HTTP gateway] required adding custom options to gRPC definitions in protobufs, and add an additional container running this reverse-proxy server." ( https://wecode.wepay.com/posts/migrating-apis-from-rest-to-grpc-at-wep… )
This still seems like a massive regression in openness to me.
dkhenry · 2020-04-30
This is a really tough problem to solve. I know Alibaba and WePay both have made solutions to try and handle it at scale
https://github.com/seata/seata
https://wecode.wepay.com/posts/waltz-a-distributed-write-ahead-log
At the end of the day I don't think anyone should be coding distributed transactions into their app's. If you need use a solution that abstracts them for you. Eventually we will get one into Vitess ( https://vitess.io ) when we have figured out something general purpose enough to work for lots of workloads
matchagreentea · 2019-07-15
vasusen · 2017-06-26
Here's a good blog post from a WePay engineer that explains some of the slowness - https://wecode.wepay.com/posts/supporting-chip-cards-at-wepay