Buildkite Blog

Even that niche has better competitors, I think. I haven't used on-prem CCI, but I used hosted CCI for several years and when my team switched to https://buildkite.com/ it was a huge breath of fresh air. I think BuildKite is the only CI system I've used I thought was actually worth paying for, and I bet it works out cheaper than self-hosted CCI in most cases as well.

It costs money, but for other readers, I've been using Buildkite[0] at two different companies across ~4 years now and it's great. It's _mostly_ self-hosted in that jobs run on your own hardware and Buildkite itself is only the job queue that your self-hosted agents pull work from. They also provide an elastic CI stack[1] so you can have auto-scaling workers running on AWS You still own enough of it that if the service is down, you can just execute the scripts manually to build, test and deploy [0] https://buildkite.com [1] https://github.com/buildkite/elastic-ci-stack-for-aws

Have you looked at https://buildkite.com/? A great UX and workflow for orchestrating tests, but the tests still run entirely in/on your own infra.

This looks great. To echo other folks’ sentiment - I think you mean CI-like pipelines specifically, though it could be extended to do some other stuff. You probably want to list some concrete use cases on the main site (e.g. CI, cron jobs, ETL?). I’m curious what differences/trade offs there are in Gaia vs something like Argo (https://github.com/argoproj/argo) or Buildkite (https://buildkite.com/). It looks like at least one difference is an actual API for steps rather than bash commands. Is there anything interesting in terms of cross job caching (e.g. saving npm install data) or persistent runners? And obviously - what else am I not thinking about?

Give https://buildkite.com/ for CI/CD a try, you'll love it!

Sure! Here goes: The App itself is boring old rails, with postgres (get me started on how much I like postgres and I will literally bore you to tears.) The original views were all haml and sass delivered via the asset pipeline. I sprinkled in angular here or there as the application grew. Today, I'm slowly tearing away the angular. I've moved to webpack for JavaScript, and React for all new UIs. I've been really enjoying both the speed of development, and how solid everything feels. Over time, I'm writing less and less ActiveRecord, and more and more sequel… still learning, but getting better. The app is run on Heroku, and its been A++ would recommend to everyone. The Calibre agent is node and websockets talking to Chrome®, theres other bits of ruby, and even some golang now. I use terraform and ansible (although, the ansible may as well have been bash scripts) through a buildkite (https://buildkite.com — the best fucking CI service on the planet) pipeline. AMIs are built, shipped to different regions, then based on some config are rolled out. Papertrail for logs - omg, logging! Essential. I run thousands of tests per day (I don't know this number off the top of my head, but its a lot)… and the nature of the internet is THERE IS ALWAYS FAILURE. Having logging at my fingertips has always made my job building Calibre easier. I think I pay $20 per month right now, and its the most important thing I pay for. -- Ok. Whew. Could probably keep going, but I'm not sure how…

Buildkite[1]. You never give them any secrets. It is essentially a frontend for agents that live on your infrastructure in the cloud, e.g. AWS. The company I work for does the CI runs on spot instances which is perfect for this kind of interruptible work. [1] https://buildkite.com/

> You can also get comparable performance out of https://buildkite.com/ which lets you self-host runners on AWS you can self-host github runners as well, with a few caveats, the most serious one being that then you are responsible for cleaning up the state of your self-hosted runner between runs https://docs.github.com/en/actions/hosting-your-own-runners/about-self… structural isolation guarantees of the form "build execution during run N cannot possibly impact build execution of run N+1" are tremendously helpful -- they reduce the number of weird CI failures and the cost to triage and fix each weird CI failure (by reducing the space of possible interactions). If you cannot offer similar guarantees when self hosting your own CI infrastructure then it may not be wise to self host.

Azure Pipelines has been really good for us. Much much more reliable than travis, and decent pricing. For self-hosted, I've heard really good things about build-kite (https://buildkite.com/) although I've not used it myself.

To be honest, if the boxes were smaller, I couldn't definitively say as I'm also red-green color blind. So I'm glad someone could clear it up for us. One great example of the case where I absolutely cannot tell is when you have a LOT of tests in something like buildkite https://buildkite.com/features and most API status pages (many small green/yello/red circles/boxes).

We have had awesome success with Buildkite [0] and self hosted build agents on EC2. We use the provided autoscale lambda and it’s pretty hands off system [1]. You also get the upside of controlling how much local caching you want to do. 0 - http://www.buildkite.com 1 - https://buildkite.com/docs/tutorials/elastic-ci-stack-aws

I haven't used Jenkins in 2 years, but before that I used the Groovy DSL plugin. A job that would run that and create the other jobs. Incidentally, a similar (but native and more streamlined) approach is taken by Buildkite: job that processes a DSL that then generates other jobs (could even be recursive if you like that sort of thing) [1] [1] https://buildkite.com

Have you looked at Buildkite[0]? The management/pipeline part is hosted (like Travis), but you run the build agents on your own infrastructure (they run on pretty much everything, including macs). Not affiliated, just a happy user. [0] http://buildkite.com

Buildkite (https://buildkite.com/) is so underrated and really nice! Much simpler interface than most other providers like Circle CI or TeamCity and works really well.

https://buildkite.com has all those features, and very easy to work with. It's a hybrid hosted model -you supply your own workers (they provide a CFN template to make that part easy), everything else (e.g. web UI/API) is hosted

https://buildkite.com We migrated there from Bamboo Cloud when Atlassian discontinued it. It's nearly a direct drop-in replacement, and they even had docs specifically about Bamboo migrations. Their customer support during onboarding was also superb, and they've been responsive to feature requests. We've been using it for almost a year now, and overall we're pretty pleased. What are you using currently?

I think that is a very valid approach. We use https://buildkite.com/ for ci and they have exactly that model, where they manage the admin panel and meta data db, but worker nodes can be deployed elsewhere

I wasn't responsible for setup, but we've been using Buildkite (https://buildkite.com) for our CI and it's been very nice. We run it through a Kubernetes cluster of workers and as far as I can tell it's been very flexible as to what's in a pipeline.

The thing that made CI more affordable for us was to rent our own Hetzner servers, and switch to https://buildkite.com Prior to this we were with CircleCI, and before that Travis CI.

I know this is off topic, but that homepage is a piece of work: https://buildkite.com I get it's quirky, but I'm at a low energy state and just wanted to know what it does... Right before I churned out, I happened to click "[E] Exit to classic Buildkite" and get sent to their original homepage: https://buildkite.com/platform/ It just tells you what it Buildkite does! Sure it looks default B2B SaaS, but more importantly it's clear. "The fastest CI platform" instead of some LinkedIn-slop manifesto. If I want to know why it's fast, I scroll down and learn it scales to lots of build agents and has unlimited parallelism! And if I wonder if it plays nice with my stack, I scroll and there's logos for a bunch of well known testing frameworks! And if I want to know if this isn't v0.0001 pre-alpha software by a pre-seed company spending runway on science-fair home pages, this one has social proof that isn't buried in a pseudo-intellectual rant! - I went down the rabbit hole of what lead to this and it's... interesting to say the least. https://medium.com/design-bootcamp/nothing-works-until-you-make-it-wor… https://www.reddit.com/r/branding/comments/1pi6b8g/nothing_works_until… https://www.reddit.com/r/devops/comments/1petsis/comment/nsm3c59/

have you tried Buildkite? https://buildkite.com

>Lack of local development. It's a known thing that there is no way of running GitHub Actions locally. This is one thing I really love about Buildkite[0] -- being able to run the agent locally. (Full disclosure: I also work for Buildkite.) The Buildkite agent runs as a normal process too (rather than as a Docker container), which makes the process of workflow development way simpler, IMO. I also keep a handful of agents running locally on my laptop for personal projects, which is nice. (Why run those processes on someone else's infra if I don't have to?) >Reusability and YAML This is another one that I believe is unique to Buildkite and that I also find super useful. You can write your workflows in YAML of course -- but you can also write them in your language of choice, and then serialize to YAML or JSON when you're ready to start the run (or even add onto the run as it's running if you need to). This lets you encapsulate and reuse (and test, etc.) workflow logic as you need. We have many large customers that do some amazing things with this capability. [0]: https://buildkite.com/docs/agent/v3

A few issues I have with this blog post: 1. It doesn't show off the unique capabilities of firecracker very well. 2. The comparison not very fair. 2a. The docker-build step (which dominates the runtime) is run without any caching, just by adding 2 lines to your build-push-action, "cache-from: type=gha, cache-to: type=gha,mode=max" you can make it a lot faster. 2b. ~1m20s of the time is just "VM start". GitHub Actions has had a rough time recently, but you should never wait that long to get your CI running in day-to-day operation. 2c. The tests are unrealistically short at 20s which allows the author to get to their 10x faster number. Let's say the GitHub Action starts in 5 seconds, the GitHub Actions cache reduces the build time to 2 minutes and the tests take 10 minutes to run. Now Firecracker is 20% faster ... You can also get comparable performance out of https://buildkite.com/ which lets you self-host runners on AWS meaning you're almost guaranteed to get a hot docker cache (running against locally attached SSDs). You can now start running your tests (almost) as fast with much more mature tooling.

The primary difference is that in Nix it's fairly easy to wrap arbitrary build systems, so you can inject third-party dependencies into your codebase without vendoring and adapting their build system (plus, through nixpkgs[0], the majority of all relevant software has already been wrapped). In Bazel you have to do a lot of that work yourself. Google has essentially ~unlimited manpower to do this for the things in their third_party tree, but for any other size of organisation this is not the case and people resort to all sorts of ugly hacks. Depending on your needs these different types of projects can also coexist with each other in Nix. To use an example from my work, we have a Nix-native build system for Go[1] and code using this[2] co-exists with code that just uses the standard Go build system[3]. Both methods end up being addressable on the same abstraction level, meaning that they both turn into equivalent build targets (you can see both in a full build[4] of our repo). And for what it's worth, some of the things Bazel gets extremely right (such as having a straightforward mapping from code location to build target, universal build command) are pretty easy to do in Nix (see readTree[5], magrathea[6]). [0]: https://github.com/NixOS/nixpkgs [1]: https://code.tvl.fyi/about/nix/buildGo [2]: https://cs.tvl.fyi/depot/-/blob/ops/besadii/default.nix [3]: https://cs.tvl.fyi/depot/-/blob/third_party/gerrit-queue/default.nix [4]:…

neat. Looks similar to https://buildkite.com .

I love buildkite and I wish they had better pricing model -- we would definitely love to pay for more. Pros of buildkite: - Very powerful. Plugin system is amazing and easy to extend. We wrote our own buildkite-docker executor, and I merged a lot of parts from it to the docker plugin upstream - UI is a lot prettier and responsive than Gitlab's - Dynamic pipelines ! - Hooks are extensive and powerful. Precheckout, Checkout, PostCheckout, precommand, command, postcommand. You got it all. - The GraphQL API is amazing, and probably one of the best API I've used in my life. - Pipelines are composable. You can make a yaml include another yaml. This was our biggest gripe with Gitlab since we use a monorepo and it got pretty big - I know for a lot of people it doesn't mean a lot, but I like emojis in the output - Agents are not stored like Gitlab. In Gitlab, once an agent connects, it gets stored in the database and never disappear until you delete it. It's problematic for our use case, because we use pre-emptibles node to save cost. It's 30% of the real price, and it dies every 24 hours. after 3 months, our Gitlab got super slow because it retained all the infos on all runners that existed. I had to make a script that deleted it from the database. - Job retries! - Docs are pretty dope - Annotate is very cool (https://buildkite.com/docs/agent/v3/cli-annotate) - Block step. Useful for confirming before deploy or something. - Pretty cheap (unlimited builds, you pay per…

Hosted service: https://buildkite.com Self hosted: Gitlab

I can't speak for CUE, but I've worked with CI and "build orchestration tools"in the past. Most CI providers provide executor APIs that let you override it as a plugin. One example is https://buildkite.com/resources/plugins/buildkite-plugins/docker-build… - you mark this as "this is using docker" and configure it in the environment, and then you provide the command. You need to be very careful about the design of the plugin, but I've done it a few times and it's viable.

Turning our snowflake infrastructure into an autoscaling fleet via a packer + terraform pipeline was one of my first projects at Buildkite back in 2016: https://buildkite.com/resources/blog/terraform-techniques-with-buildki… We still have this pipeline with terraform, but have moved from building packer images for ec2 to building containers for ecs.

Note that it can take several days or weeks for all the supported devices to get initial public builds made. You can track all the pending build jobs here: https://buildkite.com/lineageos/android/builds?branch=lineage-22.1