Slack Engineering

Funny enough, I just replied to someone who posted a link to the great lengths Slack is going through to make it use less memory on the desktop. https://slack.engineering/reducing-slacks-memory-footprint/ When the obvious answer I gave about how to reduce the memory footprint and make it more performant was “to stop using fucking [web technologies]” and create a native app But the question I always ask people who say that it’s mean old Apple keeping PWAs back, why is it that all of these same companies see a need to create an app for Android?

The Slack notification flowchart is an old favorite: https://slack.engineering/reducing-slacks-memory-footprint/

sharing some of my favorites - https://nullprogram.com/blog/2023/04/29/ - https://slack.engineering/real-time-messaging/ - https://netflixtechblog.com/migrating-netflix-to-graphql-safely-8e1e4d… - https://eli.thegreenplace.net/2023/my-favorite-prime-number-generator/

I like PHP, because it has a familiar C-like syntax and great tooling, like composer, and frameworks, like Symfony. The language has warts, but it's improving. I think its strength is its architecture, namely shared-nothing state and concurrency. [1] [1] https://slack.engineering/taking-php-seriously/

Hashes are fine, but to say that "vectors are over" is just plain nonsense. We continue to see vectors as a core part of production systems for entity representation and recommendation (example: https://slack.engineering/recommend-api) and within models themselves (example: multimodal and diffusion models). For folks into metrics, we're building a vector database specifically for storing, indexing, and searching across massive quantities of vectors (https://github.com/milvus-io/milvus), and we've seen close to exponential growth in terms of total downloads. Vectors are just getting started.

Facebook definitely does it. Thats where we first learned about this way of working. Some bigger tech companies have actually written about it (see links below) but a list of ones that we've seen with this kind of infra include LinkedIn, Shopify, Dropbox, Stripe, Slack, DataDog, and Palantir - to name a few. Eng blog links: LinkedIn: https://engineering.linkedin.com/blog/2021/building-in-the-cloud-with-… Shopify: https://shopify.engineering/modelling-developer-infrastructure-teams Slack: https://slack.engineering/development-environments-at-slack/ Doximity: https://technology.doximity.com/articles/github-codespaces-is-great-bu…

Yeah, although mostly variations on that basic premise. Airbnb Himeji: https://medium.com/airbnb-engineering/himeji-a-scalable-centralized-sy… Carta's AuthZ system: https://medium.com/building-carta/authz-cartas-highly-scalable-permiss… Slack's architecture is a bit different, but solves some of the same challenges: https://slack.engineering/role-management-at-slack/ I've also talked to a number of teams who just implemented pattern 3 internally with a custom service. Generally they've determined it's worth it to centralize all authorization data (like roles, groups, etc) into one place and perform ALL permission checks there. There are also some companies building essentially Zanzibar clones, like Auth0, Authzed, Ory Keto, and a few more.

PHP is a primary server side stack in lots of startups and unicorns. I've used it as a main server side stack in every startup I've worked in for the past 20 years. Others use it like Airbnb, Facebook, Slack, and many more. This is a good read: https://slack.engineering/taking-php-seriously/ A lot of people don't like it because of how easy it is to get into. There was similar feeling about JavaScript back in the early 2000's. People felt that it was just script kiddies that copy/pasted JavaScript so it was not respectable (at that time). BTW Laravel, and the Laravel eco-system has also helped PHP become a fantastic option for any project no matter how big or small.

> Sad that it doesn't seem to offer any deployment visibility options. I would love to connect it to our CI/CD system to see what changes are currently being rolled out. Deployment visibility is what my startup - https://sleuth.io - is trying to focus on, with light workflow sprinkled in. Other options I've found are usually tracking as a side feature of a tool like Sentry, Rollbar, or GitHub, or a completely custom app like what Slack built in https://slack.engineering/deploys-at-slack-cd0d28c61701

See this in regards to the new Slack application: https://slack.engineering/rebuilding-slack-on-the-desktop-308d6fe94ae4… It's much faster now. Also, I would be hard pressed to find a 5 sec increase in an electron app vs a native app. This article is also worth a read, focusing on why Electron fills a necessary niche: https://medium.com/@felixrieseberg/defeating-electron-e1464d075528,

Hack layered a sane type system on the existing runtime, but the changes above are all about runtime logic. These type coercions lead to many subtle bugs in PHP programs. What this demonstrates is that one can have a much safer programming language while keeping the key attributes of PHP that make it great: each web request having isolated state, concurrency model, and programmer workflow (ref. https://slack.engineering/taking-php-seriously-cf7a60065329). Yes, I'm not necessarily advocating for any web developer to start building in Hack now -- mainly because it lacks PHP's rich ecosystem. The good folks working on the PHP language may be interested in these changes though.

Without the heated tone ("entitlement in tech conversations") there's often no response to a valid complaint. We only get clear explanations from a company's engineers after a discussion is raised. > In this case, clearly there are material differences in video chat performance across browsers, or else why would Slack force Chrome support? I look forward to a post on https://slack.engineering detailing the challenges. I'm not demanding a response, it's just a wish. My decision to use a product takes certain principles into account; Slack is the kind of product that is often chosen for you by your company/open source project/community. A bit of "entitlement" — not meant as scare quotes, I just don't share the viewpoint – makes sense here, because sometimes you can't just choose not to use it.

> Would anyone recommend it Slack would, https://slack.engineering/taking-php-seriously-cf7a60065329

Flask is really good (and my preferred web framework, and full disclosure, I have forgotten how to write PHP), but it fails very badly at "no shared state," and depending on what GP meant by "a request" / "a response", it fails at that too. A Flask process handles an indefinite number of requests and responses; the built-in server handles them serially, and doing otherwise brings in either threads (and shared state) or multiple processes. The Slack article posted elsewhere in these comments https://slack.engineering/taking-php-seriously-cf7a60065329 has some excellent arguments about why PHP's approach is different in interesting ways.

I contacted support to opt out.. Here is the answer. "Hi there, Thank you for reaching out to Slack support. Your opt-out request has been completed. For clarity, Slack has platform-level machine learning models for things like channel and emoji recommendations and search results. We do not build or train these models in such a way that they could learn, memorize, or be able to reproduce some part of customer data. Our published policies cover those here (https://slack.com/trust/data-management/privacy-principles), and as shared above your opt out request has been processed. Slack AI is a separately purchased add-on that uses Large Language Models (LLMs) but does not train those LLMs on customer data. Slack AI uses LLMs hosted directly within Slack’s AWS infrastructure, so that customer data remains in-house and is not shared with any LLM provider. This ensures that Customer Data stays in that organization’s control and exclusively for that organization’s use. You can read more about how we’ve built Slack AI to be secure and private here: https://slack.engineering/how-we-built-slack-ai-to-be-secure-and-priva… Kind regards, Best regards,"

Good luck folks! I'm glad there are projects trying to solve enterprise search. I guess the main problem is the "private" aspect, if I've understood your goals correctly. Since most SaaS products lock down the private data unless you pay enterprise fees for compliance tooling. For instance, if you want to ingest data from private Slack channels or Notion groups, you have to get the users in those groups to add your bot to them, otherwise there's no way of your service getting access to the data. It's possible, just a bad UX for users. That said, built-in search for most SaaS products built after 2015 is generally quite good (e.g. Slack has an internal Learning to Rank service for a while now, which makes their search excellent: https://slack.engineering/search-at-slack/), so you'd be solving for companies like Webex and Confluence where their internal search is not great. At companies like Google they have internal search across products, which is the ideal end state, but have the benefit that they own the source code for most of their internal products.

I was Chief Architect at Slack from 2016 to 2020, and was privileged to work with the engineers who were doing the work of migrating to Vitess in that timeframe. The assumption that tenants are perfectly isolated is actually the original sin of early Slack infrastructure that we adopted Vitess to migrate away from. From some earlier features in the Enterprise product (which joins lots of "little Slacks" into a corporate-wide entity) to more post-modern features like Slack Connect (https://slack.com/help/articles/1500001422062-Start-a-direct-message-w… or Network Shared Channels (https://slack.com/blog/news/shared-channels-growth-innovation), the idea that each tenant is fully isolated was increasingly false. Vitess is a meta-layer on top of MySQL shards that asks, per table, which key to shard on. It then uses that information to maintain some distributed indexes of its own, and to plan the occasional scatter/gather query appropriately. In practice, simply migrating code from our application-sharded, per-tenant old way into the differently-sharded Vitess storage system was not a simple matter of pointing to a new database; we had to change data access patterns to avoid large fan-out reads and writes. The team did a great write-up about it here: https://slack.engineering/scaling-datastores-at-slack-with-vitess/

Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play. However, as far as I can read it, they have somewhat different views on the root cause? "Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration." vs. "The program which synced the host list generated by consul template with the HAProxy server state had a bug. It always attempted to find a slot for new webapp instances before it freed slots taken up by old webapp instances that were no longer running. This program began to fail and exit early because it was unable to find any empty slots, meaning that the running HAProxy instances weren’t getting their state updated. As the day passed and the webapp autoscaling group scaled up and down, the list of backends in the HAProxy state became more and more stale." Maybe a combination of the two?

Use the language your team is familiar with! If it is PHP don't hesitate just because it is not "cool".. PHP was a great tool back in the day and today! PHP with a decent framework like Laravel [1] or Zend Framework [2] really works well and addresses all the features that you have listed (except the last 2). Uniquely among modern languages, * PHP was born in a web server *. Its strengths are tightly coupled to the context of request-oriented, server-side execution. We've managed to develop several complex apps over the last 10 years on PHP. For Web Socket support, we use Socket.IO or Firebase and it both really well together with PHP. If you choose PHP now, you are not alone. Many companies still choose PHP for their Web Application. Slack uses PHP for most of its server-side application logic [3] [1] https://laravel.com/ [2] https://framework.zend.com/ [3] https://slack.engineering/taking-php-seriously-cf7a60065329

According to https://slack.engineering/building-hybrid-applications-with-electron/, Slack never used multiple different per-platform webviews. The earliest version of their desktop app was Mac-only and used the OS-native WebView API, but they switched to Electron at the same time they started work on making the app cross-platform. At the time, not only did Tauri not exist, but neither did the WebView2 API that it uses under the hood on Windows; they would have had to use the WebBrowser ActiveX control, which uses Internet Explorer's Trident engine and was already deprecated. So it's not so much that they rejected per-platform webviews, as that per-platform webviews were not yet really available as an option on desktop.

Based on the status report, it seems like it was related to a database disaster and mention recovering shards. Publicly they say they are using vitess[0] so I wonder if there was some issues there. I wonder if there will be a more detailed post-mortem, would be super interesting to learn more! [0] https://slack.engineering/scaling-datastores-at-slack-with-vitess/

Slack has a few eng blog posts from last year which seem to indicate they still use Hack, among other languages. https://slack.engineering/hakana-taking-hack-seriously/ describes their open source enhanced Hack type checker, which is still actively maintained (most recent commit yesterday). And https://slack.engineering/slacks-migration-to-a-cellular-architecture/ mentions they use a mix of languages: "services in the user-facing request path are written in Hack, Go, Java, and C++"

You miss-understood a critical part: https://slack.engineering/how-we-built-slack-ai-to-be-secure-and-priva… Short summary on Twitter: > Slack AI – which is our generative AI experience natively built in Slack – is a separately purchased add-on that uses Large Language Models (LLMs) but does not train those LLMs on customer data. Because Slack AI hosts the models on its own infrastructure, your data remains in your control and exclusively for your organization’s use. It never leaves Slack’s trust boundary and no third parties, including the model vendor, will have access to it. https://twitter.com/SlackHQ/status/1791276788212998503

I think you copy-pasted the wrong URL in your first link. Should be https://slack.engineering/balancing-old-tricks-with-new-feats-ai-power…

If you read the source article[0], they tried a rule-based approach first and the complexity exploded. [0] https://slack.engineering/balancing-old-tricks-with-new-feats-ai-power…

Source article: https://slack.engineering/balancing-old-tricks-with-new-feats-ai-power…

Misleading title. Maybe try this one? "Slack uses ASTs to convert test code from Enzyme to React with 22% success rate" This article is a poor summary of the actual article, which is at least linked to Slack's engineering blog [0]. [0] https://slack.engineering/balancing-old-tricks-with-new-feats-ai-power… [updated]

yyy we're in violent agreement! interesting re reddit, that's really tiny! but again, I'm even more curious about how many underlying TPS this turns into, net of rules firing, notifications and of course bots that read and analyze this comment, etc. Still, this isn't a scaling issue because all of this stuff can be done async on read replicas, which means approx unlimited scale in a single-database-under-management (e.g. here's this particular comment ID, wait for it) Slack experiences 300K write QPS: https://slack.engineering/scaling-datastores-at-slack-with-vitess/

> Integers don't scale because you need a central server to keep track of the next integer in the sequence. They most assuredly do scale. [0] Also, Slack is built on MySQL + Vitess [1], the same system behind PlanetScale, which internally uses integer IDs [2]. [0]: https://www.enterprisedb.com/docs/pgd/latest/sequences/#globally-alloc… [1]: https://slack.engineering/scaling-datastores-at-slack-with-vitess/ [2]: https://github.com/planetscale/discussion/discussions/366