Roblox Blog

See https://blog.roblox.com/ There's some impressive technology R&D. Their layered clothing system solves some hard problems. It Just Works, without the user having to fuss with mesh fit problems.

Interestingly, Roblox went down for 73 hours due to a "unique" issue with Consul as well [1]. Great read on how the issue was approached, handled, and ultimately remediated. [1] https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2…

Roblox one is one of my favourite: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2… FWIW - my company https://rootly.com/ builds a tool that helps automate postmortem write ups (e.g. timeline creation, etc).

What a great link, thanks for sharing. >"Dead tired people are more likely to make brain dead decisions" Indeed. In reading the post-mortem on the recent multi-day Roblox outage, it's hard not to imagine some bad decisions were made that only made problems worse and that likely these were on account of people just being fried by lack of sleep: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2…

Roblox had a massive 3-day outage [1] in October 2021 due to a Consul feature that didn’t work as expected. My gut with Consul is don’t use it for high-load distributed services. [1] https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2…

Roblox Oct 2021 Outage https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2…

For non-public usages, I remember Boltdb being named as one of the root causes that took down Roblox for three days! https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2…

Roblox did a very interesting and similar write up at the beginning of this year regarding a large outage they experienced at the end of 2021: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2…

Metal is preferred over Vulkan by game developers, it says a lot about how Vulkan is the one that is half baked Apple was once a supporter of Vulkan, then they abandoned the bloated ship and they made their own tech Roblox: https://blog.roblox.com/2020/05/3-years-metal/

You won't find it. Code grows, decisions get made, and practices are adopted. I am working on my fourth iteration of what turns out to be my largest undertaking. Step 1. Idea 2. Hack something 3. Commit, comment. 4. Hit a wall 5. Rewrite, commit, repeat. 6. Grow until new problems are discovered 7. Modules, tests, etc to avoid future problems. 8. New problems caused by complexity. 9. Go to 5. Read this from Roblox's outage. https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2… My recent example - I thought I could get away with 3 VMs to run my SaaS. Yes, I could, although my storage capacity guesses were 100x less than required.

The cardinal rule of databases: http://www.dbms2.com/2013/03/18/dbms-development-marklogic-hadoop/ Ignoring that leads to long postmortems about how your embedded db failed in a way you didn’t expect: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2… SQLite is good Rocksdb is good You don’t need pure go.

Does anyone know what tool this one is? https://blog.roblox.com/wp-content/uploads/2021/11/3-perf-report.png Is it really perf?

Is there any tutorial on how go get a pref report like the one show in this screenshot? https://blog.roblox.com/wp-content/uploads/2021/11/3-perf-report-1536x…

I haven’t been able to locate anything since the Halloween announcement https://blog.roblox.com/2021/10/update-recent-service-outage/ Maybe they are hoping most people forget?

There's a redirect loop between https://blog.roblox.com/2021/10/update-recent-service-outage/ and https://blog.roblox.com/2021/10/update-on-our-outage/ I'm unable to open the link and seems Wayback Machine cannot archive it either.

You can read the blog post via the... RSS feed! https://blog.roblox.com/feed/

Same, I can open https://blog.roblox.com/2021/10/ and other linked blog posts, but not this specific one.

The person mentioning secret store, and the blog post they put up does seem to somewhat point to Vault, but no name drop. > A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. This was not due to any peak in external traffic or any particular experience. Rather the failure was caused by the growth in the number of servers in our datacenters. The result was that most services at Roblox were unable to effectively communicate and deploy. https://blog.roblox.com/2021/10/update-on-our-outage/

Here's an article about ROBLOX running BERT on cpus versus GPUs. https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-daily-requ… They're not training so I think that's the difference.

https://blog.roblox.com/2008/02/the-future-of-open-source/ > I learned to program when I was 7. I started with LOGOWriter and QBASIC. What did I make? Games. It should be obvious. All kids want to write games. If your kid wants to write insurance software at age 7, you should stop wandering around aimlessly on the internet and find a good psychiatrist. Do it. Do it now. > At its heart, ROBLOX is a game development platform. You can do a lot with it without writing a line of code. But if you really get into it, you’re going to want more power. You’re going to be very motivated to figure out how to program. [...] > I don’t care what fancy private school you send your kids to. The only place your 13 year old is going to encounter a PID-Controller is in ROBLOX’s Body(Position/Velocity/Thrust) objects, which can be used to script motion for parts and models. That’s just one example.

WhatsApp stored no data and ran everything as through service, that's why they did it with so few engineers. Almost no games ever reach scale close to what you just wrote. The few that do are known with large technical teams. Same goes to everything else. The reason why you can name them almost all is because it's so rare. You are grossly overblowing the numbers. Only a very few services ever get to these numbers and once they do, they talk about the challenges of scale constantly [0] [1] [2] [0] https://blog.discordapp.com/using-rust-to-scale-elixir-for-11-million-… [1] https://blog.roblox.com/2012/04/supporting-millions-of-players-at-robl… [2] http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-fa…

Well, here's the link to the post-mortem of said "horror" story on the Roblox blog: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2… TL;DR The outage was caused by (a) they enabled a new streaming feature in Consul under unusualy high read-and-write load, (b) the load conditions triggered a pathological issue in the third-party BoltDB system upon which Consul relies and (c) all of that was exacerbated by having one consul cluster supporting multiple workloads. I'd use quotes to catalog this as a "horror" story, because this was clearly a very specific issue triggered by a specific and complex set of circumstances. The blogpost also mentions that Roblox worked closely with Hashicorp engineers to mitigate the issue, and work towards structural solutions; the post also affirms their choice to manage their infra themselves rather then moving into a public cloud solution. Sure, Kubernetes covers loads of territory. But there definitely are niches where products like Consul & Nomad do add value.

ROBLOX | Senior/Principal Software Engineers | San Francisco Bay Area | Onsite Preferred | Full Time | http://www.roblox.com The Studio Collaboration team at ROBLOX is looking for Engineers! We’re developing new ways to collaborate on art, assets, and code. We regularly release new versions and new tools to our pool of millions of active creators and directly iterate based on feedback from them. We have opportunities for Game Engine collaboration using C++ and Lua and Back End Services using C#. Join us in making collaborative game development possible for everyone! 52.2 Million Daily Active Users, 11.3 Billion Hours Engaged, 188 Countries, 32 Million Experiences, and 12 million active creators. Learn More Here - https://blog.roblox.com/ Apply Here, Principal: https://jobs.roblox.com/careers?gh_jid=3705571 Senior: https://jobs.roblox.com/careers?gh_jid=3528023 Principal Version Control: https://jobs.roblox.com/careers?gh_jid=4528630

(this got kind of long, might make it into a blog post later) > Well, in non-competitive "markets" like government services, your failure will just manifest itself in different ways. It is true that failure might manifest in different ways, but this possibly conflates not using the newest tech with failure, which isn't necessarily true. Back when the COVID tracing apps were being implemented, my country managed to mostly get it done with using a pretty old tech stack: some Ruby, some Node here and there, as well as OCI containers without anything like K8s operators or CRDs. It worked fine, it scaled decently, there were no loud snafus in the media or glaring deficiencies. I personally got to write the homepage with a pretty boring tech stack: React with no hooks, Bootstrap (instead of something like Tailwind), Nginx and Node for some read-only API components and S3 for files: https://apturicovid.lv/#en Though maybe the difference here was that it was a volunteer project by people who actually cared about what they were developing, instead of just getting a fat paycheck and padding their CVs (more about that below). Sometimes you just work at a scale where you don't need millions of writes/reads per second, given that your country literally doesn't have enough citizens for that sort of a load. You also don't always need to explore something like CQRS, pub-sub and event driven design and so on. Dare I say it, it would have probably worked just fine with Apache, PHP,…