Earthly Blog

Many have mentioned just, but I'm a much bigger fan of Earthly [0]. It allows you to write something similar to a Makefile, but everything runs in Docker. This gets you isolated builds with parallelism and caching built-in. I've found it to be great especially for small to medium projects. For some examples, I use it to publish my personal site/blog [1] and to build a C/C++/Fortran/Rust cross-compiler targeting macOS [2]. [0]: https://earthly.dev/ [1]: https://github.com/shepherdjerred/sjer.red/blob/main/Earthfile [2]: https://github.com/shepherdjerred/macos-cross-compiler/blob/main/Earth…

You want to take a look at Earthly. https://earthly.dev/ This gives you a mix of docker and a makefile. The best bit is you can test your pipeline locally and you are vendor agnostic. I'm using it here https://github.com/purton-tech/cloak

During the past few months I wrote a CI/CD pipeline using GitHub actions, Terraform, and Kubernetes. I'm not too sure Dagger would have saved that much time for me unless there was already a fully setup service already being sold. The main issue see so far is that I'd also want bringing up a CI instance of the service the same was as new prod (backup) cluster. All of that is written in Terraform already (and GitHub Actions). Why rewrite it at this point? If I need stats I could probably push them to Prometheus. So yeah, ha, sounds like just writing something like Dropbox would be easy on Linux. But it could be great if I wouldn't need to drop down to Bash Run all the time, since a lot of official or community extensions would need to be created: https://docs.dagger.io/1202/plan#plan-structure. Somewhat reminds me a bit of Earthly: https://earthly.dev/.

One primary design goal togomak had from the beginning was concurrency. All tasks run concurrently, unless a `depends_on` argument is mentioned. `just` didn't support that when I was initially building togomak, but there is a feature coming in soon which I am looking forward to: https://github.com/casey/just/pull/1562 . While I was building togomak, I read through Dagger [1], Earthly [2], Concourse CI [3], Jest and Make along with the stuff I was already working with - Jenkins, GitHub actions and GitLab CI. Dagger [1] is really great, I like its design - it supports writing pipelines in Python, Typescript, Go and a few more languages. togomak tries to abstract away a lot of it. Such as dependency management (in the case of python, the requirement of a python interpreter, and its package managers, etc). togomak is just a single statically-linked binary. [1]: https://dagger.io/ [2]: https://earthly.dev/ [3]: https://concourse-ci.org/

I recommend to try earthfiles: https://earthly.dev/earthfile This basically brings docker layer caching to CI. Only Things that changed are rebuilt and tested.

Have folks seen success with https://earthly.dev/ as a tool in their dev cycle?

This is basically what we do, except we use Earthly[1]. An Earthly satellite is basically a modified remote Docker Buildkit instance. [1]: https://earthly.dev/

Look at https://earthly.dev/ But ideally we need more info - language, env and etc. for example both CircleCI and Gradle match your current ask but something tells me you need something else

My problem with this is that it doesn't explain a lot. You can manually make a vector of a word and then step wise get up to word2vec approach and then document embedding. My post[1] does some of the first part and this great word2vec post[2] dives into it in more detail. [1] https://earthly.dev/blog/cosine_similarity_text_embeddings/ [2] https://jalammar.github.io/illustrated-word2vec/

Earthly https://earthly.dev/ Fast, consistent builds with an instantly familiar syntax – like Dockerfile and Makefile had a baby.

It seems sane. He works at Earthly, a containerized build platform: https://earthly.dev/

We are big fans of https://earthly.dev/! Although we haven't personally used Dagger, Earthly has solved our multi-service integration testing problem with elegance. Simple builds + caching baked in.

> While that doesn't guarantee success, I think it may improve my odds. Do you agree? Not your parent commenter but I wouldn't agree, no. Build systems are one of these things most companies want to setup once and want to never deal with them again, and only do so if they have zero choice. `earthly.dev` had an article some weeks ago that addressed why one of their products never took off even though it's really good and addresses grievances in the space. The TL;DR was that existing customers of other CI/CD tools did not want to migrate to a new system, especially since it doesn't offer a migration from their existing solution. Here it is, hope you find it insightful: https://earthly.dev/blog/shutting-down-earthly-ci/

This one is ridiculous. This should already exist. Until GitHub builds it, you can use GitHub Actions to kick your builds off but run them remotely on Earthly Cloud (https://earthly.dev/). Even the free tier includes arm64 remote runners. Note: I work at Earthly, but I'm not wrong about this being a good, free, arm64-native workflow for GitHub Actions.

Founder of Earthly here - besides the build debugging difficulty, I would add that modern CI/CD repeats a lot of steps: downloading, installing and configuring dependencies, making things much slower than they should be. We built Earthly [1] to tackle these two problems specifically. We're open-source (10k stars). [1]: https://earthly.dev

> https://earthly.dev/blog/golang-streamers/ This is a great list. Some of my all time favorite hackers in there such as Casey Muratori and Andreas Kling. Something that I have realized as I have gotten more experienced, is that things take time. I tried to shortcut my learning through every trick in the book and while I did manage to get pretty good in the first few years of hacking, only after I was close to 10 years of programming did I start noticing patterns and have an instinctive understanding of what to avoid and what to do. In other words taste formation is a function of building hard things over a period of a very long time. There is simply no way of short circuiting that.

https://earthly.dev/ is an interesting way to run CI workflows locally

AWK is a great tool, but I found it lacking when working with CSV files where the separator is found in quoted fields [0]. In this case, European number formatting was the problem, tripping up Sqlite's import. I resorted to pandas, where the CSV import has parameters for the thousands and decimal separators. Also see AWK HN post from 2021 [1]. [0] https://earthly.dev/blog/awk-csv/ [1] https://news.ycombinator.com/item?id=28707463

I found Dagger[1] and Earthly[2] which supposedly would solve the issue of debugging the CI locally. I haven't got time to try them out yet though. [1]: https://dagger.io/ [2]: https://earthly.dev/

I'm a big fan of Julia Evans' zines. They're short and to-the-point and are often excellent jumping off points. This one seems highly relevant: https://wizardzines.com/zines/containers/ This was also posted on HN a couple weeks ago: https://earthly.dev/blog/chroot/

> https://earthly.dev/blog/bazel-build/ That is a good read thank you.

Onedev still commits it to the repository as a file, so you’re not entirely out of luck there. Running locally is not a thing though. There’s services that do exactly that though (https://dagger.io/, https://earthly.dev/).

I think Earthly is a better replacement for make; but containerized builds are definitely slower (often to a disappointing degree). Luckily Earthly has a ton of knobs you can turn re: caching to make things suck less. https://earthly.dev

Never tried myself but this is the most serious attempt I’ve seen on alternative docker syntax https://earthly.dev/ You also have mockerfiles, being more of a proof of concept if I understand correctly https://matt-rickard.com/building-a-new-dockerfile-frontend/

Agreed. YAML is not a great format to begin with, but using it for anything slightly more sophisticated (looking at you Ansible, k8s, etc.) is an exercise in frustration. I really enjoyed working with the Earthfile format[1] used for Earthly CI, which unfortunately seems like a dead end now. It's a mix of Dockerfile and Makefile, which made it made very familiar to read and write. Best of all, it allowed running the pipeline locally exactly as it would run remotely, which made development and troubleshooting so much easier. The fact GH Actions doesn't have something equivalent is awful UX[2]. Honestly, I wish the industry hadn't settled on GitHub and GH Actions. We need better tooling and better stewards of open source than a giant corporation who has historically been hostile to open source. [1]: https://earthly.dev/earthfile [2]: Yes, I'm aware of `act`, but I've had nothing but issues with it.

> they are significant because Nix can be used to create a fully featured OS instead of just a VM Look up Bootable Containers project by RedHat [0]. Fully featured OS built from a Containerfile, bootable on bare metal. I agree that Nix design is much better than Docker, and has a bunch of features that OCI ecosystem doesn't (e.g. remote builds[1], partial downloading of the build tree, non-linear build process[2], nix store import/export, overlays, I/O isolation, much better composability), but "creating OS instead of VM" [did you mean container?] is not one of them. [0] https://github.com/containers/bootc [1] You can use DOCKER_HOST, and I'm happy that this option is there, but Nix does it better. [2] Perhaps with BuildKit it's no longer true, I haven't checked what happens if you have multi-staged build with one stage depending on multiple previous ones (which are otherwise unconnected). I think Earthly can parallelize this scenario https://earthly.dev/

> Say you don't use the helper function. Do you think you've avoided coupling? > The thing is, you're already coupled. Even if you don't use it. In the case of using multiple services, your auth service would need some changes. Now, whether those need to be done in a somewhat small service that's written in Spring Boot, launches in 5 seconds and can be debugged pretty easily due to very well known and predictable surface area, or a 10 year old Spring codebase that's using a bunch of old dependencies, needs a minute or two just to compile and launch and has layers upon layers of abstractions, some of which were badly chosen or implemented, but which you would still all need to utilize to stay consistent, making your changes take 2-5x as long to implement and then still risk missing some stuff along the way... well, that makes a world of difference. Breaking up a codebase that is a million lines big wouldn't make any of the business requirements not be there, but might make managing the particular parts a bit easier. The impact of both old code and heavily bloated codebases is so great to the point where some people hate Java because a lot of projects out there are enterprise brownfield stuff with hopelessly outdated tech stacks and practices, or like other languages just because they don't have to deal with that sort of thing: https://earthly.dev/blog/brown-green-language/ That's even before you consider it from the egoistic lens of a software dev that might want to ship…

This reads a lot like https://earthly.dev

> I just tried feeding GPT-4 some assembler as well, and while I did get it to make some optimisations, I also had to point out several mistakes it made My point of view on this is that this type of AI cannot reason, so it seems to me quite audacious to use it for programming. Following the AI's suggestion is like examining a PR from someone you don't know: don't they try to insert a backdoor or a subtle bug? Even if you use it as some kind of "rubber duck", examining its suggestions - doing the necessary reasoning and the double checks - is pretty time consuming. Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)? It feels like we are in the equivalent of programming language's "honey moon" [1] phase with this kind of technology. People will eventually get over the "wow" effect and use it for what it is good at. [1] https://earthly.dev/blog/brown-green-language/

Yeah, I got so frustrated with the odd workflow (having no sane way to locally test new/more advanced pipelines and having to do lot's of "change .gitlab-ci commits") at work that I started investigating alternatives. At home, for some hobby projects, I've been using earthly. It's just amazing. I can fully run the jobs locally and they are _blazing_ fast due to the buildkit caching. The CI now only just executes the earthly stuff and is super trivial (very little vendor lock in, I personally use woodpecker-ci, but it would only take 5 minutes to convert to use GH actions). I am not a fan of the syntax. But it's so familiar from Dockerfiles and so easy to get started I can't really complain about it. Easy to make changes, even after months not touching it. Unless I update dependencies or somehow invalidate most of the cache a normal pipeline takes <10s to run (compile, test, create and push image to a registry). This workflow is such a game-changer. It also allows, fairly easy, to do very complicated flows [1]. I've tried to get started with dagger but I don't use the currently supported SDK's and the cue-lang setup was overwhelming. I think I like the idea of a more sane syntax from dagger, but Earthly's approachability [2] just rings true. [1]: https://github.com/phoenixframework/phoenix/blob/master/Earthfile [2]: https://earthly.dev/blog/platform-values/#approachability