Aaron Maxwell

The `|| true` is often done because people use `errexit` as part of "Bash strict mode"[1], which comes with so many caveats[2] that I usually avoid it. Claude, however, loves it. [1]: http://redsymbol.net/articles/unofficial-bash-strict-mode/ [2]: https://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail

I've always had good results following "Unofficial Bash Strict Mode": http://redsymbol.net/articles/unofficial-bash-strict-mode/

Why do you find $PIPESTATUS useful? It is in theory, but in practice I've never found a situation where I wouldn't just want to fail the whole pipe if one part of it failed, which is what "set -o pipefail" is for. It's good practice to start your scripts with "set -euo pipefail" and then list exceptions to that scoped with "set +..." as needed. That's the "unofficial bash strict mode"[1]. I don't use the IFS recommendation in that article. I prefer to just keep one set of shell inanities in my head, and being used to the non-default IFS behavior doesn't help when debugging other people's shellscripts. 1. http://redsymbol.net/articles/unofficial-bash-strict-mode/

i'm surprised it doesn't recommend using set -e /other flags. [http://redsymbol.net/articles/unofficial-bash-strict-mode/] maybe it is not covered because it is not considered 'style' but i think those flags are some of the most important things you can set when writing a bash script. i guess if you are writing scripts that start other things, and then check their error codes it can be annoying because you have turn them off/on.

> I once deleted a /lib directory due to a misspelled variable in a script! Ah yes, the old "rm -rf $TEMPBASEEDIR/lib" bug. It is so easy to have that happen. Or like I did, write a script that assumes a certain environment variable is set outside the script... and then forget to set the variable before running it! To prevent these problems, use "set -u" at the top of your bash scripts. This will halt the script instead of silently inserting an empty string if you use an undefined variable. "set -e" (halt on nonzero return code) and "set -o pipefail" (propagate error returns through a pipe) are also useful options, or combine them with: set -euo pipefail Here's a great article that describes these options and some tips for using them: http://redsymbol.net/articles/unofficial-bash-strict-mode/

It's cool, nice work! And thank you for sharing it. This is just spitballing, you've already solved problem you set out to and get full marks. There are always many different ways to do things with bash/shell. Here, you could probably do the entire thing in AWK, which you seem to already be aware of. This might be more readable or it might not. If this were a script someone wrote for more general usage, I'd recommend to use the unofficial "bash strict mode"[0], and make sure to check that the awk/sed calls succeed. Shell has a nasty habit of limping along as far as it can if something goes wrong along the way and it's very easy to just end up with bad output and not notice until it's too late. Another thing I only recently discovered is envsubst, a tool that's commonly available and tailor-made for substituting variables. You could pipe a template string or pile into that instead of the repetitive calls to echo. What you have here is fine and instantly readable, but if it were three times as long it might not be so nice. Here you could use awk to extract the variables and envsubst to put them into your template. Honestly I don't think shell is a good language for writing perfect, robust things anyway. But it's good for exactly what you're doing, though, writing something for yourself that solves a problem without spending too much time on it (although it's fine to spend a lot of time on it if you're learning and/or just enjoying yourself). [0]…

It usually doesn't matter much, but there are some situations where it can matter a lot. For one thing, you can't use seek() on a pipe, so e.g. `cat bigfile | tail` has to read through the entire file to find the end, but `tail bigfile` will read the file backward from the end, completely skipping the irrelevant beginning and middle. With `pv bigfile | whatever`, pv (which is basically a pipeline progress indicator) can tell how big file is and tell you how for through you are as a percentage; with `cat bigfile | pv | whatever`, it has no idea (unless you add a flag to tell it). Also, `cat bigfile | head` will end up killing cat with a SIGPIPE signal after head exits; if you're using something like "Unofficial bash strict mode" [1], this will cause your script to exit prematurely. Another sometimes-important difference is that if there are multiple input files, `somecommand file1 file2 file3` can tell what data is coming from which file; with `cat file1 file2 file3 | somecommand` they're all mashed together, and the program has no idea what's coming from where. In general, though, I think it's mostly a matter of people's expertise level in using the shell. If you're a beginner, it makes sense to learn one very general way to do things (`cat |`), and use it everywhere. But as you gain expertise, you learn other ways of doing it, and will choose the best method for each specific situation. While `cat |` is usually an ok method to read from a file, it's almost never the best…

I agree with some of your frustrations, but it's honestly not that bad once you setup your environment properly and get used to some of the quirks. > I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Adopt some best practices[1] and have a template to start new scripts with. ShellCheck also helps with avoiding the most common pitfalls. > I find myself reaching for purpose-built tools rather than ad-hoc pipelines It helps to distinguish day-to-day interactive work in the shell from writing portable shell scripts that will be robust and require little maintenance for years to come. For the former you want user friendly tools that help you the most with a particular task. For the latter you want to use battle-tested tools with a stable interface that are available on many systems or are preferably built-in to the shell. > I am getting more and more frustrated at basic shell mechanics like history handling That's mostly a solved problem in many shells. Zsh supports it natively[2]. I wouldn't want to discourage you from writing your own shell, good luck with that, but there are plenty of good POSIX compatible and alternative shells out there. We can agree that "real" programming languages are much more powerful, safer and friendlier, but plain old shell scripts are the right tool for the job in many situations and shouldn't be ignored at all cost. [1]:…

On one hand, if you're managing a few servers with shell scripts, you'll invariably end up with a framework of sorts. On the other hand, after a few years playing with bash and posix shell -- I'm fairly certain I'd prefer bash, to posix shell -- it seems rather unlikely you can easily use the same set of scripts for BSD, open solaris and various linux distributions -- if your target is linux, pretty much all distributions ship with bash in base install (just remember to use proper "bang" of #!/usr/bin/env bash or equivalent). And shell script is a fickle language -- I'd at least consider using "trap" and/or "set -e", "set -u" -- maybe "set -p" rather than "echo 'what I am about to do'; what I am about to do". See eg: http://redsymbol.net/articles/unofficial-bash-strict-mode/ I agree that ansible is far from perfect - but I'm not convinced posixcube is a great alternative right now. From a brief look at the code, it does look nicely structured and commented, though! I've played a bit with the idea of doing something similar (mostly because I'm not convinced all the complexity present in ansible is necessary for my use case -- which appear to be similar to your use-case -- a few small simple server configs). I've seen people here on HN recommending some makefiles for deployment (along with simple shell scripts) -- and I've played a bit with using m4 for templates rather than here-documents -- and it looks like that might work fine. Does mean your…

Shellcheck is a godsend https://github.com/jamespwilliams/strictbash, I wrote this little wrapper a while back that you can use as a shebang for scripts. It runs shellcheck for you before the script executes, so it’s not possible to run the script at all if there are failures. It also sets all the bash “strict mode” [0] flags. [0] http://redsymbol.net/articles/unofficial-bash-strict-mode/

There's quite a few situations where it does NOT exit when you'd expect it to. This is one of those. What you're seeing there is a variant of the "Short-Circuiting Considerations" discussed here: http://redsymbol.net/articles/unofficial-bash-strict-mode/ Basically, when you short circuit, it will not cause the script to exit. However, the short circuit still affects the return value of the function, or the script itself if it's the last line of the script. So your short circuit is causing the function to return non-zero, which then causes the script to exit.

A better approach is to avoid writing bash scripts. They're just too dangerous, even with "strict mode" (http://redsymbol.net/articles/unofficial-bash-strict-mode/). Better to use Python, or Ruby, or a Rust or Go executable.

Also, "unofficial bash strict mode" is terrific, especially if you intend to distribute a script: http://redsymbol.net/articles/unofficial-bash-strict-mode/ set -euo pipefail IFS=$'\n\t' (You can always turn, for example, error checking back off for an individual portion of the script with `set +e`)

Also known as "bash strict mode". It's a godsend if you're unlucky enough to have to deal with bash scripting. http://redsymbol.net/articles/unofficial-bash-strict-mode/ EDIT: The comic strip would be better in three rows of two panels - a row for each set flag.

This looks fun. I can't imagine it being directly useful but the idea itself is fun. Couple of immediate issues with the repo: - Sample output in the README should not be prefixed with a "$". That's only for the shell command prompt. - Use ``` blocks in your README file. It's easier to format multi-line text. - Don't use $RANDOM, it's not secure. Use mktemp: https://github.com/alcover/runc/blob/73aaecea5a4cd357f7172dd1618d98f0b… - Don't exit with success if the command failed: https://github.com/alcover/runc/blob/73aaecea5a4cd357f7172dd1618d98f0b… - Use Bash strict mode: http://redsymbol.net/articles/unofficial-bash-strict-mode/

related: http://redsymbol.net/articles/unofficial-bash-strict-mode

+1 This is where I’ve learned proper bash. People always compliment my bash skills, but the only thing I do is sticking to the wooledge wiki’s rules + using the unofficial bash strict mode (http://redsymbol.net/articles/unofficial-bash-strict-mode/).

Good idea -- here is an explanation: http://redsymbol.net/articles/unofficial-bash-strict-mode/

This is generally terrible advice. A better option is to 'set -e' and ensure that the bash script exits when there's a failure. Bash scripts can't be idempotent because they operate in an external environment that can't ever be. The better option is to just be extra safe. See: > http://redsymbol.net/articles/unofficial-bash-strict-mode/

Obligatory: http://redsymbol.net/articles/unofficial-bash-strict-mode/

Author of article here (but not submitter). I wrote this years ago, and it's fun to see it pop up on HN like this every year or two! Here's another bash article people like: http://redsymbol.net/articles/unofficial-bash-strict-mode/

I like scala too but I have to say, it is a bit on a heavy side when it comes to scripting. re: the rm -rf "$STEAMROOT/"* there is much simpler way to avoid it: set -euo pipefail IFS=$'\n\t' This would prevent this bug and many others. For whole overview on that see: http://redsymbol.net/articles/unofficial-bash-strict-mode/

Chances are that every system this script runs on has "mktemp," and trapping exit makes it easy to clean up when things are finished. https://www.mktemp.org/manual.html http://redsymbol.net/articles/bash-exit-traps/

This article has some good points, and I was glad to learn about the "trap" command, to clean up temp files on unexpected exit. There is also a little overlap with this "unofficial bash strict mode," which I now use all the time: #!/bin/bash set -euo pipefail IFS=$'\n\t' --- http://redsymbol.net/articles/unofficial-bash-strict-mode/

The bash -e and -u options might have saved him here: http://redsymbol.net/articles/unofficial-bash-strict-mode/

I have always tried to avoid using functions in my bash scripts. That said, it's always nice to other people's bash programming techniques and style. This one had the biggest impact for me, although I can't imagine it's news to any of you on here: http://redsymbol.net/articles/unofficial-bash-strict-mode/

The thing I wished I had learned earlier is "quick and dirty assertions". If you write lots of functions in Bash, you quickly end up getting tripped up by cases where an argument is omitted and the function does something totally batshit given the missing (empty string) argument. Now, the canonical way to handle this is to put validators on your input, (and make sure those validators don't crash with cryptic errors if someone calling your function is using "set -u") like so: function() myfunc { local foo="${1:-}" if [ -z "$foo" ]; then echo "Invalid first parameter!" >&2 return 127 fi ... } ...but man, that's time consuming when you have lots of parameters. Instead, the quick and dirty way is to just "assert" via [parameter expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Exp… function myfunc() { local foo="${1:?First parameter must be provided}" ... } Much quicker, especially when throwing things together in a hurry. It has a gotcha, though: ":?" assertion doesn't cause a function to return early, it shuts down the whole interpreter after outputting the error. So it's more like a true assert() statement than an input validator. If you'd only ever call your function in a subshell, this won't matter (because the subshell will exit early with a nonzero code, big deal), but otherwise it can be a nasty surprise to users when an argument-validation issue inside a function shuts the program…