Dagger Blog

> GHA has "service containers", but unfortunately the feature is too basic to address real-world use cases: it assumes a container image can just … boot! … and only talk to the code via the network. Real world use cases often require serialized steps between the test & the dependencies, e.g., to create or init database dirs, set up certs, etc.) My biased recommendation is to write a custom Dagger function, and run it in your GHA workflow. https://dagger.io If you find me on the Dagger discord, I will gladly write a code snippet summarizing what I have in mind, based on what you explained of your CI stack. We use GHA ourselves and use this pattern to great effect. Disclaimer: I work there :)

Dagger (https://dagger.io) is a great way to use BuildKit through language SDKs. It's such a better paradigm than Dockerfiles, I cannot imagine going back. Dagger is by the same folks that brought us Docker. This is their fresh take on solving the problem of container building and much more. BuildKit can more than build images and Dagger unlocks it for you.

Hey, https://dagger.io employee here. We can help with this. I'd recommend taking a look at out Python SDK (https://docs.dagger.io/sdk/python) to build your pipelines and then leverage Dagger Cloud (https://dagger.io/blog/dagger-cloud) for fast production deployments. Happy to help if you have any questions.

I'd encourage anyone thinking of using make to look at alternatives. Make is great, but is quickly becomes a ball of duct-tape. Make works very well when you spend the time to express your dependency tree, but realistically that never happens and people tend to add hacks upon hacks for Makefiles. Not only that, but they don't scale well as your project adds more components, such as integration testing, documentation, etc. I found Earthly[0] to be a great replacement. Everything runs in Docker. Your builds are reproducible, cache-able, and parallelizable by default. I've heard Dagger[1] is another good tool in the space. [0]: https://earthly.dev/ [1]: https://dagger.io/

One primary design goal togomak had from the beginning was concurrency. All tasks run concurrently, unless a `depends_on` argument is mentioned. `just` didn't support that when I was initially building togomak, but there is a feature coming in soon which I am looking forward to: https://github.com/casey/just/pull/1562 . While I was building togomak, I read through Dagger [1], Earthly [2], Concourse CI [3], Jest and Make along with the stuff I was already working with - Jenkins, GitHub actions and GitLab CI. Dagger [1] is really great, I like its design - it supports writing pipelines in Python, Typescript, Go and a few more languages. togomak tries to abstract away a lot of it. Such as dependency management (in the case of python, the requirement of a python interpreter, and its package managers, etc). togomak is just a single statically-linked binary. [1]: https://dagger.io/ [2]: https://earthly.dev/ [3]: https://concourse-ci.org/

This is fascinating. It checks a lot of my boxes. I was recently looking at ActionsFlow [0] which is similar but runs on GitHub Actions. My thoughts - 1. I don't see proper secret storage being handled. You typically don't want your API keys in your code. What would you recommend instead? 2. "OAuth" based secrets. Many integrations require giving access to an App via OAuth, which involves a flow. I think that's being handled internally from the video [1] and from this project [2], but it's not clear. How is that handled? A common use case I'd automated once is that when a GitHub project gets starred, the developers public information is scrapped and they are then followed on Twitter, if their twitter handle is found. With Trigger.dev, the twitter part isn't clear. 3. Error Handling - What about when some job fails to run? I understand there is a delay mechanism. But what about injecting custom error handling? Sending a message on slack, for example. 4. Dashboards - They look awesome. And I get the impression that each "action" in the code is mapped to individual blocks in the dashboard. I'd love to be able to see a proper graph of the flow. I love that I can see the json request / response for each. It'll make debugging easier when some API changes or fails. 5. No Code solutions - In the long run, I can easily picture writing the integration I want in plain text, and having Github CoPilot or ChatGPT generate the code for me, and then I can quickly modify it. 6.…

You're 100% right IMHO about the convergence of powerful CI pipelines and full build systems. I'm very curious what you'll think if you try Dagger, which is my tool of choice for programming the convergence of CI and build systems. (Not affiliated, just a happy customer) https://dagger.io/

Dagger (https://dagger.io) recently seems to have reinvented/rebranded itself as some llm agent platform.

People don’t know about https://dagger.io/, which solves this.

I commend anyone who’s taking a hard look at our current CI/CD practices. Good work! Succinctly stating the problems is easier said than done. I believe https://dagger.io checks all these manifesto boxes and more. At least that’s where I’m focusing my attention.

Sounds a lot like https://dagger.io/ . Maybe you can take some inspiration from them

Check out https://dagger.io/. Write declarative pipelines in code, reproducibly run anywhere.

https://dagger.io/ may be an alternative. You can run it in GitHub actions somehow as well.

Another https://dagger.io fan here. Have been using it since late 2021 to continuously deploy a Phoenix app to Fly.io: https://github.com/thechangelog/changelog.com/pull/395. Every commit goes into production. This is what the GHA workflow currently looks like: https://github.com/thechangelog/changelog.com/blob/c7b8a57b28ee4b74716… FWIW, you can see how everything fits together in this architecture diagram: https://github.com/thechangelog/changelog.com/blob/master/INFRASTRUCTU…

We've been using Jsonnet to generate Pods for our complex DAGs at Opsgenie, wrote a similar one using Tekton: https://mustafaakin.dev/posts/2020-04-26-using-jsonnet-to-generate-dyn… But if I was doing it again now, I would just use https://cuelang.org/ or https://dagger.io Jsonnet is really hard to debug.

I find this really annoying. There's also no concept of a package like Helm Charts which means you can distribute all your images on Docker Hub but then when it comes to compose files and configurations with templating your up @$%^ creek. I was hoping something like https://dagger.io/ might solve this but they currently don't support compose files, or maybe https://www.acorn.io/ ? It does feel like there is a legitimate space for a multi-container service application standard that isn't just kubernetes, unless perhaps things like k3d and docker desktop end up so good at running tiny k8s that docker compose dies.

This doesn't have everything you mention, but I recently learned about https://dagger.io/ which allows you to write your pipelines as code (Go, Python, etc) and then run them anywhere in containers. I always hated that in order to test my pipelines, I actually have to trigger it on some special machine (the CI server).

I found Dagger[1] and Earthly[2] which supposedly would solve the issue of debugging the CI locally. I haven't got time to try them out yet though. [1]: https://dagger.io/ [2]: https://earthly.dev/

Onedev still commits it to the repository as a file, so you’re not entirely out of luck there. Running locally is not a thing though. There’s services that do exactly that though (https://dagger.io/, https://earthly.dev/).

Can Gitlab CI use CircleCI workflows? There's less risk in just changing your VCS platform, rather than both your VCS and CI/CD at once, especially if you have to rewrite all your workflows. Obviously something like Dagger[1] solves this, but I don't know how widely used that is at this point. [1] https://dagger.io/

I’m hopeful that https://dagger.io will help CI “shift left”! Especially the ability/easiness to run it locally, and the built in caching/DAG

Dagger sounds like it will satisfy your requirements. Note, however, that it’s still very early days for that project. See: https://dagger.io/

Looks similar to me with https://dagger.io/ but with lisp instead of cue. I’m looking forward to see where this space goes.

Is this similar to Dagger[1] ? [1] https://dagger.io

Is this tool similar to https://dagger.io/?

I misread it as Dagger, the CI/CD tool (https://dagger.io)

> These alternatives are far more familiar and easier to work with. You can get most (but not all) of the benefits of NixOS with far less work. > Worse is better. There are some potential alternatives out there that seem inspired by Nix but which are less strict. I think a system that offers and manages Nix's purity but doesn't require it could end up the final winner some day. Shea Levy had some ideas for eventually achieving this in Nix that he called 'Nomia' (probably riffing on the name of Sander van der Burg's Dysnomia, I'd guess), but his life circumstances changed before the project really materialized. But Nix still could be the place where that is first achieved. Meanwhile new container image-building DSLs like Dagger¹ and Bass² look a lot like a Nix which is impure by default, and bash builders are replaced with BuildKit directives. And there's Denxi³, which looks more like a Nix that lets you reduce its purity when you want to. For now, though, the Nix ecosystem is still vastly more capable than these alternatives because of all the incredible work that Nixpkgs represents. Nix we know it today could still end up the concrete basis for what succeeds it, and not just an inspiration. 1: https://dagger.io/ 2: https://bass-lang.org/ 3: https://docs.racket-lang.org/denxi-guide/

I support the effort to build a platform-agnostic CI/CD pipeline solution, but I don't want it in the form of yet another platform. Rather it needs to be a protocol that any platform can tie in to. I'm especially wary since this is another VC-backed effort that will eventually need to be monetized in some shape or form. Additionally, as someone else here has already mentioned, my mind first went to Dagger, the dependency injection tool (https://dagger.dev). That tool in particular was named as a play on DAG (directed acyclic graphs), whereas in this case I don't think it would apply since there may be instances where you'd want cycles in a pipeline. On a whim, I clicked on "Trademark Guidelines" (https://dagger.io/trademark) and from that page alone I would recommend avoiding this based on the aggressive language used to try and claim ownership of generic words. According to their own language, it seems I'm violating their guidelines by writing this comment. > Our Marks consist of the following registered, unregistered and/or pending trademarks, service marks and logos which are subject to change without notice: Dagger; Blocklayer; and other designs, logos or marks which may be referred to in your specific license agreement or otherwise. > Blocklayer does not permit using any of our Marks ... to identify non-Blocklayer products, services or technology Which would include Dagger, the dependency injection tool. Other sections of note: > Do Not Use As Nouns (This one…

dagger was originally CUE-based, but there was not enough demand so it was dropped. https://dagger.io/blog/ending-cue-support