My company is currently adopting this and I don't see the appeal yet - likely from a lack of knowing much about it.
I added it to a side project just to get familiar and it added quite a few sdk files and folders to my project, and lots of decorators. It also required Docker and yadda yadda yadda.
I just could not justify using it compared to just running some regular Typescript file with Bun (or, in a different project, `go run cmd/ci/main.go`)
Suppose I give you two functions f, and g. Can you run f(g()) without breaking things? The honest answer is you don't know until you read the functions, which is a slow and difficult thing to do.
Suppose I give you functions f and g of respective types int -> str and Nothing -> str. Can you compose them? No, and you see this immediately from the types. Types make reasoning about composability a lot easier.
Of course, it's not a panacea, and it's less helpful the more side effects a function has. Can we compose pure int->int functions? Of course! Can we compose two of them where the second expects some image to exist in some docker registry? You'll need to read the first to be able to tell.
Given the highly side effectful nature of pipelines, I'd think the applicability of types would be limited. But maybe that's just a lack of imagination on my part.
Certainly information like "this pipeline expects these variables" and "this pipeline sets these variables" are susceptible to a typed approach, and it would make things easier. By how much, I don't know.
Not sure but my guess is because they aren't a good fit for many languages. If you need a task runner then often languages will have a built in option or there are better alternatives than Make. If you need a build system then Make isn't a good fit for a lot of modern languages.
I’ve been using earthly a lot lately and its general value prop is simple: it turns out that if Buildkit is your primary build tool that Make targets can almost always be represented as OCI image layers. The killer feature IMO is that its syntax is familiar enough to end users of both Make and Dockerfiles that engineers tend to be willing to onboard to it. A lot of these other solutions that use proprietary DSL’s struggle to cover every use case, and the implementations in turing complete typical language SDK approach often forces you into analysis paralysis if there is no existing pattern.
My struggle with Make and bash is that they're not very expressive - maybe that's something we want in our CIs, but I've always preferred writing an actual program in that program's native language for CI/CD, even if it has to shell out some commands every now and again.
Maybe none, but at some point you may want to do other things at buildtime, such as generating an sqlite db or generate code stubs for protobuf. Having a universal, and highly refined, tool like Make will help developers without domain knowledge. It also does not exclude the use of other tool like Just and Docker. A Makefile is also an easy jump-off point for a build pipeline.
Then introduce it for those things. But a makefile to call go build, go test, docker push, Ecs update-services provides no value other than all of a sudden not working on windows without another tool.
At that point you end up with a makefile that has a 1:1 mapping with targets in my experience.
At a previous job, we had an enormous makefile, most of which was defining phone targets and translating make arguments into maven arguments. All the actual targets were calling maven. Make provided no value at all in that, other than requiring you to know
Make and maven to modify anything in the build.
Personally I’d rather a shell script for a command runner in most cases
The `just` tool is a better and much easier to understand command runner than `make`, however. Much less feature surface, too, which eliminates nasty surprises coming from the unnecessary complexity of `make`.
`just` is 90% similar to `make` in syntax, only it has 100x less foot guns. :)
Also I'll never understand the appeal of "not having to install a tool". We're not in the 1980s anymore when that was an actual chore. You run a command, the tool is there (including in CI/CD), boom, done. What am I missing here?
Bootstrapping can be painful in some languages or frameworks. Not everyone is running containerised builds where there are ephemeral environments that you just install a tool (and pay the 30+ second cost per build to run apt-get update). There’s certainly value in having a front door entry point. But I think it should be a shell script, not a makefile.
Yes to your last. Either sh/bash script or a precompiled Golang program. If installing a tool is really such a problem then having a precompiled strongly typed program doing various tasks should be a no-brainer.
I started openly hating `make` because I re-learned its specifics and quirks several times over the course of 10-ish years and then figured that I want to learn stuff with a staying power in my brain. I don't use `make` every work day so eventually any quirks disappear -- that's how our brains work.
So that's why I learned most of `just` and it hasn't betrayed me so far, not once. Though I did write a few Elixir and Golang programs for running various tasks in production environment, too.
I have a hot take on this. I don’t care how you build and deploy as long as it’s reproducible and the whole process can be tracked in their metadata. I’d rather have a process validating CI/CD stages and artifacts metadata in a central db than unifying pipelines that won’t get standardized due communication complexity. This way I can have a conversation on visibility rather than code edge cases.
> The Fix: Use a full modern programming language, with its existing testing frameworks and tooling.
I was reading the article and thinking myself "a lot of this is fixed if the pipeline is just a Python script." And really, if I was to start building a new CI/CD tool today the "user facing" portion would be a Python library that contains helper functions for interfacing with with the larger CI/CD system. Not because I like Python (I'd rather Ruby) but because it is ubiquitous and completely sufficient for describing a CI/CD pipeline.
I'm firmly of the opinion that once we start implementing "the power of real code: loops, conditionals, runtime logic, standard libraries, and more" in YAML then YAML was the wrong choice. I absolutely despise Ansible for the same reason and wish I could still write Chef cookbooks.
> Pipeline definitions are scattered across multiple tools—GitHub Actions, Jenkins, ArgoCD, Kubernetes—and environments. This fragmentation leads to confusion, configuration drift, and duplicated effort.
So are they talking about some sort of meta language compiling into multiple yaml configs for the different environments or a single separate CI tool that has plugins and integrates with GitHub/gitlab/etc?
I do agree with them about the need for a real programming language. I hate yaml in gitlabs config, it is very hard to interpret how it will be interpreted. Things were much easier when I was scripting Jenkins even though I didn't know or like groovy then with gitlab
I had a look at the example glu deployment pipeline and I’m decidedly unimpressed.
Admittedly most of my criticism is related to the choice of Go as an implementation language: more than 80% of the code volume is error handling boilerplate!
Before the lovers of Go start making the usual arguments consider that in a high-level pipeline script every step is expected to fail in novel and interesting ways! This isn’t “normal code” where fallible external I/O interactions are few and far between, so error handling overhead is amortised over many lines of logic! Instead the code becomes all error handling with logic… in there… somewhere. Good luck even spotting it.
Second, I don’t see the benefit of glu (specifically) over established IaC systems such as Pulumi — which is polyglot and allows the use of languages that aren’t mostly repetitive error handling ceremony.
This seems like an internally developed tool that suits the purposes of a single org “thrown over the fence” in the hope that the open source community will contribute to their private tool.
I commend anyone who’s taking a hard look at our current CI/CD practices. Good work! Succinctly stating the problems is easier said than done.
I believe https://dagger.io checks all these manifesto boxes and more. At least that’s where I’m focusing my attention.
My company is currently adopting this and I don't see the appeal yet - likely from a lack of knowing much about it.
I added it to a side project just to get familiar and it added quite a few sdk files and folders to my project, and lots of decorators. It also required Docker and yadda yadda yadda.
I just could not justify using it compared to just running some regular Typescript file with Bun (or, in a different project, `go run cmd/ci/main.go`)
> Without types, it is difficult to compose pipelines together.
I would gladly hear this argument expanded. It's really not obvious to me that that's the case.
Suppose I give you two functions f, and g. Can you run f(g()) without breaking things? The honest answer is you don't know until you read the functions, which is a slow and difficult thing to do.
Suppose I give you functions f and g of respective types int -> str and Nothing -> str. Can you compose them? No, and you see this immediately from the types. Types make reasoning about composability a lot easier.
Of course, it's not a panacea, and it's less helpful the more side effects a function has. Can we compose pure int->int functions? Of course! Can we compose two of them where the second expects some image to exist in some docker registry? You'll need to read the first to be able to tell.
Given the highly side effectful nature of pipelines, I'd think the applicability of types would be limited. But maybe that's just a lack of imagination on my part.
Certainly information like "this pipeline expects these variables" and "this pipeline sets these variables" are susceptible to a typed approach, and it would make things easier. By how much, I don't know.
Is any of this reproducible? Not sure why that requirement has been quietly overlooked.
I've worked in this space for a long time and can't make head or tail of what glue is.
A motivating examplt would be help which I might have missed?
I'm thinking before we build a CI/CI pipeline, make sure there is a Makefile. Why have they gone out of style?
Not sure but my guess is because they aren't a good fit for many languages. If you need a task runner then often languages will have a built in option or there are better alternatives than Make. If you need a build system then Make isn't a good fit for a lot of modern languages.
I’ve been using earthly a lot lately and its general value prop is simple: it turns out that if Buildkit is your primary build tool that Make targets can almost always be represented as OCI image layers. The killer feature IMO is that its syntax is familiar enough to end users of both Make and Dockerfiles that engineers tend to be willing to onboard to it. A lot of these other solutions that use proprietary DSL’s struggle to cover every use case, and the implementations in turing complete typical language SDK approach often forces you into analysis paralysis if there is no existing pattern.
My struggle with Make and bash is that they're not very expressive - maybe that's something we want in our CIs, but I've always preferred writing an actual program in that program's native language for CI/CD, even if it has to shell out some commands every now and again.
I prefer that, too. I've heard (less-experienced) tech leads forbid Makefiles because they're not declarative enough compared to yaml.
If I want to build and test a golang app and push it to a container repository, what value does a makefile provide over go build && docker push?
All the tools do their own dependency tracking already (unfortunately).
Maybe none, but at some point you may want to do other things at buildtime, such as generating an sqlite db or generate code stubs for protobuf. Having a universal, and highly refined, tool like Make will help developers without domain knowledge. It also does not exclude the use of other tool like Just and Docker. A Makefile is also an easy jump-off point for a build pipeline.
Then introduce it for those things. But a makefile to call go build, go test, docker push, Ecs update-services provides no value other than all of a sudden not working on windows without another tool.
Honestly I like it just to keep it as a command runner with the needed flags. Then in the unholy YAML there's just make build, make test, etc
At that point you end up with a makefile that has a 1:1 mapping with targets in my experience. At a previous job, we had an enormous makefile, most of which was defining phone targets and translating make arguments into maven arguments. All the actual targets were calling maven. Make provided no value at all in that, other than requiring you to know Make and maven to modify anything in the build.
Personally I’d rather a shell script for a command runner in most cases
Yeah, mine often just defers back to some shell scripts. But it's useful to enumerate them in the Makefile.
The `just` tool is a better and much easier to understand command runner than `make`, however. Much less feature surface, too, which eliminates nasty surprises coming from the unnecessary complexity of `make`.
There are a pletora of tools better than Make. But it's a standard tool, everyone is familiar with it, you probably don't even have to install it.
`just` is 90% similar to `make` in syntax, only it has 100x less foot guns. :)
Also I'll never understand the appeal of "not having to install a tool". We're not in the 1980s anymore when that was an actual chore. You run a command, the tool is there (including in CI/CD), boom, done. What am I missing here?
The advantages you list are flimsy at best.
Bootstrapping can be painful in some languages or frameworks. Not everyone is running containerised builds where there are ephemeral environments that you just install a tool (and pay the 30+ second cost per build to run apt-get update). There’s certainly value in having a front door entry point. But I think it should be a shell script, not a makefile.
Yes to your last. Either sh/bash script or a precompiled Golang program. If installing a tool is really such a problem then having a precompiled strongly typed program doing various tasks should be a no-brainer.
I started openly hating `make` because I re-learned its specifics and quirks several times over the course of 10-ish years and then figured that I want to learn stuff with a staying power in my brain. I don't use `make` every work day so eventually any quirks disappear -- that's how our brains work.
So that's why I learned most of `just` and it hasn't betrayed me so far, not once. Though I did write a few Elixir and Golang programs for running various tasks in production environment, too.
I have a hot take on this. I don’t care how you build and deploy as long as it’s reproducible and the whole process can be tracked in their metadata. I’d rather have a process validating CI/CD stages and artifacts metadata in a central db than unifying pipelines that won’t get standardized due communication complexity. This way I can have a conversation on visibility rather than code edge cases.
This is important for SBOM (software bill-of-materials) which will soon be mandatory in regulated domains.
I'm guessing this is relevant: https://news.ycombinator.com/item?id=42267316 Show HN: Glu – Deployment pipeline framework as code - Nov, 2024 - 2 comments
And, tellingly, it seems they still haven't provided a "why not ${other tool}" anywhere that I can readily spot
You beat me to it: why not Dagger?
FTA:
> The Fix: Use a full modern programming language, with its existing testing frameworks and tooling.
I was reading the article and thinking myself "a lot of this is fixed if the pipeline is just a Python script." And really, if I was to start building a new CI/CD tool today the "user facing" portion would be a Python library that contains helper functions for interfacing with with the larger CI/CD system. Not because I like Python (I'd rather Ruby) but because it is ubiquitous and completely sufficient for describing a CI/CD pipeline.
I'm firmly of the opinion that once we start implementing "the power of real code: loops, conditionals, runtime logic, standard libraries, and more" in YAML then YAML was the wrong choice. I absolutely despise Ansible for the same reason and wish I could still write Chef cookbooks.
Best pipeline I’ve had the pleasure to design is AWS CodePipeline via AWS CDK. Ticks all boxes. Uses pure TypeScript code.
> Pipeline definitions are scattered across multiple tools—GitHub Actions, Jenkins, ArgoCD, Kubernetes—and environments. This fragmentation leads to confusion, configuration drift, and duplicated effort.
So are they talking about some sort of meta language compiling into multiple yaml configs for the different environments or a single separate CI tool that has plugins and integrates with GitHub/gitlab/etc?
I do agree with them about the need for a real programming language. I hate yaml in gitlabs config, it is very hard to interpret how it will be interpreted. Things were much easier when I was scripting Jenkins even though I didn't know or like groovy then with gitlab
I had a look at the example glu deployment pipeline and I’m decidedly unimpressed.
Admittedly most of my criticism is related to the choice of Go as an implementation language: more than 80% of the code volume is error handling boilerplate!
Before the lovers of Go start making the usual arguments consider that in a high-level pipeline script every step is expected to fail in novel and interesting ways! This isn’t “normal code” where fallible external I/O interactions are few and far between, so error handling overhead is amortised over many lines of logic! Instead the code becomes all error handling with logic… in there… somewhere. Good luck even spotting it.
Second, I don’t see the benefit of glu (specifically) over established IaC systems such as Pulumi — which is polyglot and allows the use of languages that aren’t mostly repetitive error handling ceremony.
This seems like an internally developed tool that suits the purposes of a single org “thrown over the fence” in the hope that the open source community will contribute to their private tool.
None of these are a problem anymore since the advent of Nix.