Sven Mascheck

I think colon at the start of a line was used for labels for goto statments. https://www.in-ulm.de/~mascheck/bourne/PWB/goto.1.html DESCRIPTION Goto is allowed only when the Shell is taking commands from a file. The file is searched from the beginning for a line beginning with `:' followed by one or more spaces followed by the label. If such a line is found, the goto command returns. Since the read pointer in the command file points to the line after the label, the effect is to cause the Shell to transfer to the labelled line.

The thing that surprised me was that you can't write an interpreter in an interpreted language, at least not in obsd. It is possible if you jump through a few hoops but you can't directly call it. An example: if you made a language in python /bin/my_lang: #does nothing but pretend it does #!/usr/local/bin/python3 import sys print('my_lang args', sys.argv) for line in sys.stdin: print('invalid_line:', line) my_script: #!/bin/my_lang line of stuff another line of stuff chmod u+x my_script ./my_script Probably for the best, but I was a bit sad that my recursive interpreter scheme was not going to work. Update: looks like linux does allow nested interpreters, good for them. https://www.in-ulm.de/~mascheck/various/shebang/#interpreter-script really that whole document is a delightful read.

I like the notion of 'set -e' and at the same time I hate it. First, because it behaves inconsistently across shells/versions [1] and second, because it doesn't always work as expected. For example, when you depend on the 'set -e' behavior in a function and call the function from within a condition, the 'set -e' has no effect at all. So you better don't count on 'set -e'. But don't expect me to follow my own advice, as not using 'set -e' isn't a good option either... [1] https://www.in-ulm.de/~mascheck/various/set-e/

> 100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters. In UNIX file names, there is exactly one unusual character: ASCII NULL (or '\0'). Every other character is usual, including spaces, newlines, tabs and other historical ASCII control characters. ------ > No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long. Shell does not impose length limits, the UNIX kernel does by defining the limit of how much can be passed into the «execve» syscall which UNIX shells use to create new processes. You can find the length limit on your system by running «getconf ARG_MAX» in the shell as it varies across different systems (it varies even across different versions of the same system; in Linux, it is now reportedly 1/4th of ulimit -s). The command line limit is calculated using following constituents: command line limit = «environment size (env | wc -c)» *plus* «(ARG_MAX *minus* environment size» *minus* «POSIX recommended 2048»). How the length limit works is explained at length (please pardon the pun) here: https://www.in-ulm.de/~mascheck/various/argmax/ ------ > Excellent path parsing. […] finding the file extension […] File names in UNIX do not have extensions, they simply have names. Is «.bashrc»: 1) a full and complete file name or 2) an empty file name with the «bashrc» extension? It is (1).…

http://www.in-ulm.de/~mascheck/various/echo+printf/ is the raw data version. POSIX defines echo as only taking string parameters and no options, but notes that behaviour facing `-n` is implementation-defined. BSD and GNU echo implement `echo -n` as not printing a trailing newline, but `echo` commonly calls to a shell builtin which may or may not follow that behaviour (and may switch behaviour depending on whether the shell is in "sh mode" or not), so `echo -n` could print `-n<newline>` or nothing whatsoever (empty string and suppressed newline) depending on the utils set, the shell, and the shell's runmode. GNU echo also supports -e, -E, --version and --help options, and much like -n shells and other utils set may or may not support these. For instance on my machine (OSX 10.11) * zsh (builtin) interprets -e, -E and -n as options (but not version or help) * bash (builtin) also does, except when invoked as sh in which case it does not and all parameters are literal (this may also apply to zsh) * dash (builtin) interprets -n, but none of the others. bash note may also apply to it. * BSD echo interprets -n but will print -e, -E, version and help literally * GNU echo interprets all of the above So if you use echo with any non-literal parameter, or with one of the parameters listed above, in a script you distribute to un-controlled third-parties as an sh script (rather than e.g. a bash or zsh script specifically) you will suffer from portability issues. And that's just…

You can run this pup to find out what's underneath! https://www.in-ulm.de/~mascheck/various/whatshell/

> Honestly, I wish I could ask the person who designed it, why they made such decisions. https://web.archive.org/web/20250115051355/https://www.bell-labs.com/u… https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf https://www.in-ulm.de/~mascheck/bourne/index.html#origins They also include citations to papers by Thompson, Bourne, and others.

> Using set -e ensures that your script exits immediately if any command fails, preventing cascading errors. There's a huge asterisk around using -e https://www.in-ulm.de/~mascheck/various/set-e/ contains a matrix of the different shells (and versions) and how they handle different conditional orders with -e

Oh and another one I use all the time is as an "identity transform" when selecting between filters, e.g. dostuff () { if [[ $1 = clean ]]; then grep -v dirt else cat fi | do_other_stuff } See also https://www.in-ulm.de/~mascheck/various/uuoc/ (via https://lobste.rs/s/rtvp2u/useful_uses_cat#c_0xpqkr )

I wonder what system allows a 4GB string to be passed as a command. A brief look at https://www.in-ulm.de/~mascheck/various/argmax/ makes me think none by far? It seems this limit is at most 2MB which is nowhere near 4GB. Am I misunderstanding something?

Nitpick: The $(…) syntax was not invented by GNU, it was invented by Kenneth Almquist at the University of Washington. https://www.in-ulm.de/~mascheck/various/ash/#original

Yeah I guess I read that too quickly -- sorry. This: #!python3 Seems to work (i.e., search $PATH) on macos but AFAIK nowhere else. It fails on both ubuntu20 and centos8. Per this: https://www.in-ulm.de/~mascheck/various/shebang/, relative paths are supposed to be interpreted as relative to ., so I'm guessing that is why everyone uses /usr/bin/env. Given that shebang is implemented in the kernel (and not in user-space like coreutils) I think env is more likely to change than the shebang mechanism, though who knows -- there have been recent changes to shebang so perhaps further work on this will help: https://lwn.net/Articles/779997/.

I do not think that _this_ was or will be ever an issue to myself or any system that i have to deal with and i do not see any use for it but i guess somebody else did. Anyway I believe this page is a very nice read on the background and a little history https://www.in-ulm.de/~mascheck/various/shebang/

Everyone talks about the macro abuse, nobody talks about the memory management abuse: https://www.in-ulm.de/~mascheck/bourne/segv.html > In comp.arch, 05/97, <[email protected]>, John Mashey writes: > For speed, Steve B had used a clever trick of using a memory arena without checking for the end, but placing it so that running off the end would cause a memory fault, which the shell then trapped, allocated more memory, then returned to the instruction that caused the trap and continued. The MC68000 (in order to go fast) had an exception model that broke this (among other things) and caused some grief to a whole generation of people porting UNIX to 68Ks in the early 1980s.

> ARG_MAX, maximum length of arguments for a new process > or why do you get > command: arg list too long > [...] > It's only the exec() system call and its direct variants, which will yield this error. They return the corresponding error condition E2BIG (<sys/errno.h>). The shell is not to blame, it just delivers this error to you. https://www.in-ulm.de/~mascheck/various/argmax/ Additionally, see also https://unix.stackexchange.com/a/120842 which speaks a bit about Linux specifics, and how Linux behaves a bit differently from a lot of the other operating systems in the Unix family with regards to limits on the length of arguments.

Bourne shells as early as Version 7 Unix (1979, [1]) had rather elaborate interpolation of variables into strings (i.e., not just ${foo} but ${foo?bar}, etc.). I'm reasonably sure that earlier versions of sh had some facility for this as well. So, it's not really a PHP innovation, although some people might have seen it first there. [1] https://www.in-ulm.de/~mascheck/bourne/v7/ -> "Parameter substitution"

Apparently Unix Version 6 (1975) used Thompson Shell which (provided the manuals are correct [1][2]) didn't support arbitrary variables (and even environment variables were not a thing back then). However for use in scripts $0, $1, etc were replaces with the script name and the arguments. 1: https://www.in-ulm.de/~mascheck/bourne/v6/ 2: https://etsh.nl/man/_tsh.1.html

OSX/macOS, minix and for a time FreeBSD[0] [0] https://www.in-ulm.de/~mascheck/various/shebang/ (See table near bottom)

Original HN title had the word "Google" in it. According to this document, Google requires Bash. I do not use Bash. I use Almquist shell. I try to avoid using uncommon "features" or utilities. I am not a frequent Linux user but whenever I have to use it, all of the hundreds of scripts I wrote using another OS and Almquist shell still work. They all work in Bash. Over the years, I used this site as a reference: https://www.in-ulm.de/~mascheck/

https://www.in-ulm.de/~mascheck/various/shebang/#env mentions other OSes that have only /bin/env, although admittedly they are all quite old.

An OS that allows shebang scripts to have setuid or capabilities ends up allowing security holes, as seen in traditional Unix variants; see http://www.faqs.org/faqs/unix-faq/faq/part4/section-7.html and https://www.in-ulm.de/~mascheck/various/shebang/#setuid Therefore, Linux simply doesn't allow it.

https://github.com/skarnet/execline/raw/master/src/execline/elglob.c https://github.com/skarnet/execline/raw/master/src/libexecline/exlsn_e… Simple. http://www.in-ulm.de/~mascheck/various/argmax/ execlineb -c 'elglob a /*/*/*/* ls $a' (statically-linked execlineb) If I am not mistken, ARG_MAX will be the limit. Straightforward.

There was a separate goto utility in early UNIX. Pretty sure you can still find the source code if you look. Meanwhile, it's mentioned here: http://www.in-ulm.de/~mascheck/bourne/

They might propagate to subshells. Someone did a lot of checking: http://www.in-ulm.de/~mascheck/various/set-e/

Yes. It was the inspiration for the IOCCC (International Obfuscated C Code Competition). http://www.ioccc.org/faq.html But the macros were the least of its problems: It allocated memory by trapping SIGSEGV (SIGnal for SEGmentiation Violation; that is, the signal the OS sends a program when it's tried to access memory beyond the region it owns); the function which caught SIGSEGV allocated more RAM. This made it difficult to port sh to the Motorola 68000-based computers which were the first generation of Unix workstations. http://www.in-ulm.de/~mascheck/bourne/segv.html https://news.ycombinator.com/item?id=8843951

Nope, there are systems in which it's in /bin/env, though you have to go fairly exotic before you find one that doesn't at least have a symlink in /usr/bin. I don't believe POSIX says anything about it http://www.in-ulm.de/~mascheck/various/shebang/#env In fact, POSIX doesn't even standardize the shebang at all; it is simply mentioned as a possible extension.

Beware that using another script as the executable in shebang may not work everywhere [0] [0] http://www.in-ulm.de/~mascheck/various/shebang/#interpreter-script

more/older https://www.in-ulm.de/~mascheck/various/shebang/

https://news.ycombinator.com/item?id=35768615 Perhaps dash now has these features but does NetBSD sh or FreeBSD sh have them. What's the point of "pure sh" if it's restricted to specific versions of shells. Opinions may differ but I'd rather learn the "lowest common denominator". I use the same scripts on both Linux and BSD so I focus on portability. I do not use bash for scripting because it's larger and slower. One source I consult is https://www.in-ulm.de/~mascheck/bourne/ It is sometimes submitted to HN, but not IMO enough https://news.ycombinator.com/from?site=in-ulm.de I'm using the latest version of dash but I still don't think it has these ternary operators. Even if it has them, I'm not going to use them in scripts unless I know they'll work on older dash versions, NetBSD sh, FreeBSD sh, etc. Needless to say the process of updating these shells is generally very slow. It takes years. "Pure sh" implies there is some advantage over "bashisms". I thought maybe it was portability but who knows.

dash is an Almquist shell. At this point there are several major ones (the Debian one, the FreeBSD one, the NetBSD one, and the BusyBox one) and a whole raft of minor ones (Minix, Cygwin, Android, et al.). * https://www.in-ulm.de/~mascheck/various/ash/ And Wikipedia does not say that. It says that the Minix Almquist shell, specifically, had a non-conformant test utility. The Almquist shell did not, after all, have a built-in test command at all to start with, so the standard conformance of that utility wasn't a matter for the Almquist shell. Once again Wikipedia is wrong, because Thomas Dickey's original page that purportedly supports this claim points out that the Minix test command was an external command and not part of the Almquist shell. M. Dickey even pointed to the source code for the external command on GitHub. * https://invisible-island.net/autoconf/portability-test.html#portabilit… * https://github.com/Stichting-MINIX-Research-Foundation/minix/commits/m… * https://commons.wikimedia.org/wiki/File:Don't%20abbreviate%20as%20…