Google Cloud Platform Blog

My personal upvote for the Google Cloud Platform blog: https://cloudplatform.googleblog.com/ ...a pretty mixed bag with some product & PR posts inside, but the gems inside (especially SRE / CRE life lessons) are pretty awesome. Also, we're getting started ourselves with some quality tech material - not too much there yet, but our Debugging Postgres post got a lot of love from the community: https://www.justwatch.com/blog/

I'm not with GCP but I'm fairly closely connected to its technical staff and user community and a heavy consumer and have been so for about 4 years. It gives me a somewhat unique perspective where I'm happy to give them lip when they need it and they have at times and they also ask for my input fairly frequently. In my opinion they are all in on GCP and it's definitely going nowhere but up. They've been on massive hiring sprees and it's to the point that when I visited them at the Googleplex a few months ago a bunch of their campus buildings were now labeled as Google Cloud buildings. Obviously logos can be removed and maybe it will say Android/Allo/whatever in 3 years but the team is incredible and they seem laser focused on growing the platform. You can go through their blog https://cloudplatform.googleblog.com/ and the improvements they're making and the pace at which they're making them is pretty great. I've used them for ~4 years so I've definitely had frustrations, especially prior to them reorganizing their Support/Account Management structure about 6-12 mo ago (I do k8s so I interact with this side of things as infrequently as possible). This is a huge money maker for them and I think they're doing a great job albeit they still need to focus on their soft skills (support, billing pains, etc). I have migrated a few companies from AWS to GCP and they've all been happy. If the day ever comes to leave GCP I have no problem making that call; I'm not handcuffed to…

Khanacademy is great and don't get me wrong, but what I see is an engineering blog that doesn't force HTTPS and an ad for Google products. All that Khanacademy did was optimize code, setup the partner console properly and pay the (likely enormous) bill. As I read in another comment here: "How to scale: Make it somebody else's problem" + pay the bill. Edit: ah, a case study from Google about Khanacademy. This post was definitely an ad: https://cloud.google.com/customers/khan-academy Another: https://cloudplatform.googleblog.com/2013/08/khan-academy-runs-on-goog…

GCP has Serverless Containers coming soon: https://cloudplatform.googleblog.com/2018/07/bringing-the-best-of-serv… Zeit V1 supports launching containers (but isnt part of V2): https://zeit.co/docs/v1/getting-started/deployment/#docker-deployment

See also: https://medium.com/@mchmarny/build-deploy-manage-modern-serverless-wor… https://www.ibm.com/blogs/cloud-computing/2018/07/24/ibm-cloud-google-… and one by this "Google" thing: https://cloudplatform.googleblog.com/2018/07/bringing-the-best-of-serv…

Thanks for sharing and very insightful. Guess the TPUs are the real deal. About 1/2 the cost for similar performance. Would assume Google is able to do that because of the less power required. I am actually more curious to get a paper on the new speech NN Google is using. Suppose to be 16k samples a second through a NN is hard to imagine how they did that and was able to roll it out as you would think the cost would be prohibitive. You are ultimately competing with a much less compute heavy solution. https://cloudplatform.googleblog.com/2018/03/introducing-Cloud-Text-to… Suspect this was only possible because of the TPUs. Can't think of anything else where controlling the entire stack including the silicon would be more important than AI applications.

https://cloudplatform.googleblog.com/2015/03/Google-Compute-Engine-use… is probably the easiest to follow.

We are very far from trusted hardware. It's very easy to imagine what they would do - step one is helping American networks use the best practices - use open-source software, keep dependencies up-to-date, have bug bounties to find vulnerabilities in popular frameworks, etc. Step two would be to consider some shared infrastructure, probably subcontracting with a cloud provider (AWS/Azure/GCP), hopefully multiple. Once we get to that step, then you can start considering things like Google's Titan (https://cloudplatform.googleblog.com/2017/08/Titan-in-depth-security-i… But there's a lot of low hanging fruit before we get there.

Google's network (especially w/ BBR[1]) is amazing and this makes the price point more approachable for other use cases (like running your own CDN[2]). [1] https://cloudplatform.googleblog.com/2017/07/TCP-BBR-congestion-contro… [2] https://blog.edgemesh.com/deploy-a-global-private-cdn-on-your-lunch-br…

[I'm a Google employee, opinions are my own] The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a great way to survive a sudden spike in traffic, but not optimal, as serving errors is also bad (unless your clients have some sort of retry mechanism in-place). But if your clients do have a retry mechanism in place (especially one that has a backoff time on retries), it can be a great way to handle high loads. [0] https://cloudplatform.googleblog.com/2016/12/using-load-shedding-to-su…

> What do they know about the CAP theorem that I don't? Their statement, for what it's worth: https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Spanner-and-…

But that's exactly what Google's Customer Reliability Engineering addresses. CRE says to the enterprise customer, if you're willing to bring your software up to Google's standards, then the guy answering the 4am call when downstream screws up will be some of those same engineers who give Google its legendary reliability, and not one of the enterprise's engineers with little experience on GCP. Those standards are the difference between a hack backed up by sales engineers and "premier support" and a tech manager who never finds out that anything was amiss because all the failovers just worked. https://cloudplatform.googleblog.com/2016/10/introducing-a-new-era-of-… FWIW I work in one of these places with crazy institutional conservatism

Not sure if you are looking for a commercial solution, but we (Twistlock [1]) develop a security suite for enterprises working with Docker and / or Kubernetes. In fact, we are officially recommended by Google for working with GKE which is very much based on Kubernetes [2]. I'd be glad to elaborate if relevant. [1] https://www.twistlock.com/ [2] https://cloudplatform.googleblog.com/2015/11/enhancements-to-Container…

> If you can create one platform service that developers love and runs on cheaper specialized hardware, you'll be able to destroy your competitors and signal an alternative hardware arms race. I think this was what Google is trying with their TPU for running TensorFlow for machine learning applications. https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machi…

Not that AWS isn't great overall--just that compared to https://cloudplatform.googleblog.com/2016/02/Compute-Engine-now-with-3… they are not competive for low cost and high performance storage. (And their internal networking isn't so great either for smaller instances)

Excellent point Miles (and as you know I work in GCP too), I would add https://cloudplatform.googleblog.com/2015/05/AppScale-and-App-Engine-W… and https://www.youtube.com/embed/2fRTgosMtw4?rel=0&controls=0&sho… (in both of which, of course, you feature quite prominently:-)...

[Disclaimers: I worked on Borg and Omega, and currently work on Kubernetes/GKE. Everything here is my personal opinion.] There's a lot to unpack here, but I'll do my best. I don't see Kubernetes locking people into GKE. There's an extensive conformance program (https://github.com/cncf/k8s-conformance) administered by the CNCF. AWS and Azure both have certified hosted Kubernetes offerings. Portability is in Google's best interest. Go, Docker, and etcd were the best open-source technologies for the job at the time Kubernetes was created (and arguably still are). Open-sourcing Borg would have been impossible, due to its use of many Google-specific libraries (though a number of those have been open-sourced since then), and its close coupling to the Google production environment. Commenting more specifically on each of the pieces you mentioned: * Go was chosen over C++ because, like C++, it is a systems language, but is much more accessible for building an open-source community. * Docker was (and still is) by far the most popular container runtime, and the slimmer containerd makes it even more appropriate to serve as the container runtime for a system like Kubernetes. While it's true that in Borg the container runtime and "package" (container image) management systems are separate, the tradeoffs between packaging more in the image vs. pre-installing dependencies on the host are exactly the same as with Docker images. In any event, it's very feasible to build very slim Docker…

Feature development of GAE (especially the Standard Environment) never stopped. Some of the features require multi quarters of work (e.g. https://cloudplatform.googleblog.com/2018/05/Increase-performance-whil… some are more enterprise-focused, some are not user visible but we improve the reliability with a new stack behind the curtain. We are changing the engine while flying (pun not intended). For various reasons, supporting new runtimes was never an easy task, even for small version bumps. This has changed drastically, starting with Java8 runtime, to the recently announced NodeJS beta, and to the closed alpha of Python 3 (remember this is Python 3.7). If you followed recent announcements from Google you probably can connect the dots. I think we are pretty open on what we are working on to our customers, as long as an NDA is signed. If you are interested be sure to let our cloud support team know! Disclaimer: I'm on App Engine SRE team.

This looks like a helpful approach to a tricky topic! I'm curious how the post-deploy migrations work when you run all your migrations at once, from the beginning, e.g. in Circle CI or when onboarding a new developer? Also, do you do anything to protect yourself against migrations written at time t_0 and then run much later at time t_n? I've seen a lot of problems there when migrations use application code (e.g. ActiveRecord models), and then that code changes. (My solution is to never call model code from migrations, but there are other ways.) This isn't really specific to your article I guess, but does your approach making managing that harder? Easier? Does having that details table help at all when you have migrations on a long-lived branch that is merged after other migrations have been added? Or is the solution there just to rebase the branch and test before merging it in? EDIT: I think this blog post by the Google SRE folks is great reading for people thinking about migrations and deployment: https://cloudplatform.googleblog.com/2017/03/reliable-releases-and-rol… I've never been comfortable with rolling back migrations in production, but their plan changed my mind that it can be done safely. Is your approach compatible with theirs?

If you find yourself impacted by these changes in your home lab (and chances are you will; taking ballooning out of Free is nothing but petty), keep in mind that you can follow the lead of the major hosting companies and jump ship to KVM via oVirt or some homebrew. KVM is better in a number of ways anyway, one of them being a lack of control by Citrix, not to mention when Linode switched they got ~30% more perf out of Linodes. Xen has been dying for a while. EC2 is heading off it, and they've held out the longest. This change is evidence that Citrix would like to accelerate that trend by killing off the home lab users that stuck it out (read: me). Looks like I have a new Christmas project. https://cloudplatform.googleblog.com/2017/01/7-ways-we-harden-our-KVM-… https://www.theregister.co.uk/2017/11/07/aws_writes_new_kvm_based_hype… https://blog.linode.com/2015/06/16/linode-turns-12-heres-some-kvm/ https://www.digitalocean.com/help/technical/general/ (Cmd+F KVM)

Google's backbone is a little bit different. Google Cloud shares this backbone with Youtube, Maps, and the rest. Not only are there DC-to-DC cables, our backbone extends to a vast number of Edge POPS (more than AWS and Azure combined), detailed at [0]. Our DC-to-DC network is pretty great too, allowing things like Spanner to exist (minimizing P in CAP etc) at [1]. Here's what it means for you in practice: - Google's Load Balancer supports a single global anycast endpoint. - When your packet tries to hit Google network, it hits the nearest POP, which acts as an on-ramp to the network and traverses only Google network, not touching public. - Similarly, when data is en route to a customer, Google network will take it all the way to the nearest DC or POP on its private backbone. - Google by default gives you a global software-defined VPC. No need to create VPN tunnels between zones/regions/etc. (work at G) [0] https://peering.google.com/#/infrastructure [1] https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Spanner-and-…

Publish those results? It'd be very interesting to see. And, it sounds like you think there are benchmarks missing from the existing common set of things people are measuring -- what's a very specific network you'd like to see added to the mix? VGG16 doesn't fall into my radar of "modern and applicable" in the days of ResNet. Using NCCL is great; TF now supports it, as of about a month and a half ago (though I don't know how tightly integrated it is): https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contri… From the benchmarks available, and not knowing what your in-house experiments show, I don't believe that the "internal network stack" is key to scaling. The scalability numbers shown on tensorflow.org/performance are very reasonable: From 902 images/sec to 1783 (1.97x) going from 32->64 K80 GPUs on Amazon for Inception v3, and 565->981 (1.7x) for ResNet-512. I'd love to be proved wrong. That 1.7x scaling on ResNet-512 would be a great point of comparison, for example. From my student Hyeontaek's results, I actually suspect that there are scheduling improvements that could make up some of that difference, not networking improvements. As I'm sure you know, of course, and are just fishing for, the reason that code links against gRPC externally is because trying to extract Google's internal networking code from the full internal software codebase would be ridiculous. I think it's far more likely to see the other…

I'm still rooting for KVM because it has the best hardware compatibility, it in mainline and its performance is the best. However the security situation with QEMU is not as rosy as some of you portray it (and its not as bad as it looks either). The average customer/user is going to look at the number of QEMU CVEs on cvedetails and compare them to the number of Xen's and go with the latter. However most of the QEMU bugs only affect legacy emulated hardware or components not enabled in a KVM guest, most won't think that far. It would clear things up if you have a table on your site showing which QEMU vulnerabilties affect a specified default configuration of a RHEL/Debian guest out of the box in libvirt. See this for example: https://www.qubes-os.org/security/xsa/ What I want to see: * Adoption of QEMU-lite as the default mode for Linux guests. There's no point to running Linux in almost any emulated hardware. * A builtin monitoring solution like Google has that detects excessive DRAM bitflips [1] and cache misses [2] and terminates the guests to foil rowhammer and covert channel attacks. * A re-design of KSM thats not prone to rowhammer abuse [3] * [1] https://cloudplatform.googleblog.com/2017/01/7-ways-we-harden-our-KVM-… [2] https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_… [3] http://www.cs.vu.nl/~kaveh/pubs/pdf/ffs-usenixsec16.pdf

That's fair - random person on the internet, I get it. I have common usernames on the various services you know and love, so feel free to dig about. As for transparency, there is a big push to increase the trust in Google Cloud. We're a big company, and it's hard to get that personal connection to build the trust. I think our blog is trying to get this info out, here is a good recent one: https://cloudplatform.googleblog.com/2017/01/how-we-secure-our-infrast… My opinion is that Google is a very honest company. These whitepapers are our attempt to accurately describe what steps we are taking, in good faith, to ensure Google Cloud is the best, and most secure place, to do your computing on. As a personal anecdote, I once worked for Google (and quit, and came back) about 8 years ago. The difference in how secretive we are is night and day. I can say the word 'borg' in public, and even tell you what it is. And this is accelerating under Cloud, because we know that big enterprises aren't going to use us without a lot of details and assurances.

San Francisco - Streak (https://www.streak.com/careers) | Profitable and growing | Tech Leads (Backend, Mobile and Front-End) | ONSITE | H1-B Transfers welcome Here are the hard problems you'll work on here at Streak: Sales Autopilot We want to do for salespeople what autopilot did for pilots. We will suggest to sales professionals the actions they need to take next to close deals. They'll instead need to monitor our "autopilot" system and manually override only when needed. Unlike most AI startups, we can actually build this - we have 100M+ datapoints of sales professionals manually closing deals to make the training actually possible. Email Graphs We're building systems to better understand how organizations communicate with one another. Email is currently siloed into individual user inboxes, we're building a graph database containing every email message in our users' inboxes to give them insights into how their company communicates with their customers. Reverse Engineer Gmail We strive for deep integration with Gmail, because that's where our users live. Our integration has to be seamless, deep, performant, resilient and secure. It's such a hard problem, even our competitors use our tech (InboxSDK). We're doing it next for Inbox, Calendar, and more. Outside of the hard problems you'll also have a very large impact, not just on our organization but on our customers. We have the best of both worlds: we're small (6 eng) so no meetings and other big company BS, but…

San Francisco - Streak (https://www.streak.com/careers) | Profitable and growing | Tech Leads (Backend, Mobile and Front-End) | ONSITE | H1-B Transfers welcome Here are the hard problems you'll work on here at Streak: Sales Autopilot We want to do for salespeople what autopilot did for pilots. We will suggest to sales professionals the actions they need to take next to close deals. They'll instead need to monitor our "autopilot" system and manually override only when needed. Unlike most AI startups, we can actually build this - we have 100M+ datapoints of sales professionals manually closing deals to make the training actually possible. Email Graphs We're building systems to better understand how organizations communicate with one another. Email is currently siloed into individual user inboxes, we're building a graph database containing every email message in our users' inboxes to give them insights into how their company communicates with their customers. Reverse Engineer Gmail We strive for deep integration with Gmail, because that's where our users live. Our integration has to be seamless, deep, performant, resilient and secure. It's such a hard problem, even our competitors use our tech (InboxSDK). We're doing it next for Inbox, Calendar, and more. Outside of the hard problems you'll also have a very large impact, not just on our organization but on our customers. We have the best of both worlds: we're small (6 eng) so no meetings and other big company BS, but…

If you recall AWS didn't scale instantly originally either. We had super bursty traffic, and had to go with Google Cloud (very early days! [0]) because you'd need to communicate with AWS and pre-warm the ELB capacity of your expected bursts. We did a dead launch to 60 million customers (0 to 60 million, no organic growth phase) this way. I wouldn't want to do that on a VPS. [0] https://cloudplatform.googleblog.com/2013/11/?m=1

This definitely happened with Google Maps, where Google was clearly the dominant player. Has it happened to other services? It seems far less likely to happen for business cloud services where they are a distance 2nd (3rd?) to AWS. I know of some examples (admittedly ancient) where they have reduced costs: https://cloudplatform.googleblog.com/2015/05/Pay-Less-Compute-Moore.ht…

It seems Google Cloud Storage introduced it in August 2013[1], enabled by default. [1]: https://cloudplatform.googleblog.com/2013/08/google-cloud-storage-now-…

This blog post is pretty old (2015) but gives a good introduction. https://cloudplatform.googleblog.com/2015/03/Google-Compute-Engine-use…