The thing that surprised me was that you can't write an interpreter in an interpreted language, at least not in obsd. It is possible if you jump through a few hoops but you can't directly call it.
An example: if you made a language in python
/bin/my_lang: #does nothing but pretend it does
#!/usr/local/bin/python3
import sys
print('my_lang args', sys.argv)
for line in sys.stdin:
print('invalid_line:', line)
my_script:
#!/bin/my_lang
line of stuff
another line of stuff
chmod u+x my_script
./my_script
Probably for the best, but I was a bit sad that my recursive interpreter scheme was not going to work.
Update: looks like linux does allow nested interpreters, good for them.
Worked for me, but the way you described it has issues:
1. You chmod my_script twice.
2. Did you chmod u+x /bin/my_lang too? Since you put it in /bin, are you sure the owner isn't root?, in which case your user wouldn't have execute permission. Try +x instead of u+x.
3. Do you have python in that path? Try `/usr/bin/env python` instead.
4. In case you expected otherwise, my_script wouldn't be passed through stdin. It's just provided as an argument to my_lang.
I think that was what I was trying to figure out, how the program was passed. but OpenBSD does not do nested interpreters, it looks like if I had tried Linux it would have worked.
NixOS is annoying because everything is weird and symlinked and so I find myself fairly frequently making the mistake of writing `#!/bin/bash`, only to be told it can't find it, and I have to replace the path with `/run/current-system/sw/bin/bash`.
Or at least I thought I did; apparently I can just have done `#!bash`. I just tested this, and it worked fine. You learn something new every day I guess.
Because /bin is the standard location for bash. The only one that breaks that expectation is NixOS (and maybe GuixSD?), apparently. I'm surprised they didn't symlink /bin or put a stub. Last time I tried NixOS was like 10 years ago. I thought there was a /bin/bash, but maybe it was just a /bin/sh?
Other interpreters like python, ruby, etc. have more likelyhood of being used with "virtual environments", so it's more common to use /usr/bin/env with them.
/bin is the "standard" location for bash on a subset of Linux distributions and basically no other Unix...
So it's not really a standard.
/bin/sh is a much more common convention but once again, not a standard.
There really isn't a truly portable shebang, but the same can be said about executables themselves. As part of the build or install step of whatever thing you're making, you should really be looking these up and changing them.
It's only really NixOS as far as I know that doesn't ever put bash in /bin/bash (as far as Linux distributions go). But, on the other hand, there are quite a few distros (or at least flavours of distros) which don't ship bash by default (alpine, minimal versions of most distros, and embedded-Linux focused stuff if you count it). I imagine the most common "installation" of Linux is userspace in a container (yeah I know there's no kernel there, but nobody who talks about "Linux" broadly speaking specifically cares about the kernel) and a good chunk of those will be minimal with no bash.
Bash has to be explicitly installed on OpenBSD, FreeBSD, NetBSD (I think, haven't used it in a while) and probably a bunch of others. And in all of those cases (that I know of) it doesn't end up in /bin/bash once installed.
The default bash shipped on macs is so abhorrently ancient that it would be strictly better if it didn't exist because it would reduce the number of people who think bash scripts I write are broken (they're not broken, they just inevitably depend on some bash 4+ feature). Moreover, hardcoding /bin/bash as your shebang in this case will prevent anyone from remediating this problem by installing a non-ancient bash because the old one in /bin/bash will still get used.
Will not wok on OpenBSD where the shell that comes with the system is ksh at /bin/ksh and /bin/sh and if you want bash it is a third party package and correspondingly gets installed as /usr/local/bin/bash
It does get awkward, especially when porting. all your third party libraries and includes are in /usr/local/lib /usr/local/include but at least it is better than freebsd which also insists on putting all third party configs under /usr/local/etc/
They do symlink /bin/sh to be fair, and that's very often good enough for a lot of scripts. That's what I usually do if I don't need anything bash offers.
Thing is, a few years ago when Debian changed its default sh from bash to ... either ash or dash, I forget which, I got into the habit of always writing `#!/bin/bash` at the top of my scripts, in case I didn't realize that something I was using was a bashism not found in classic /bin/sh. So if I used Nix (I don't, since for my particular use cases the juice isn't worth the squeeze), I would get seriously messed up by that.
It's guaranteed to work provided that Bash is in the path.
It's very common for Python. Less so for Bash for two reasons: because the person who writes the script references /bin/sh instead (which is required to be there) even when they are writing bash-isms, or because the person who writes the script assumes that Bash is universally available as /bin/bash.
/bin/sh is NOT required by POSIX, they explicitly warn that it may not exist[1].
> Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH , ensuring that the returned pathname is an absolute pathname and not a shell built-in.
I don't know if this is still a thing, but I distinctly remember when playing around with NixOS years ago that `env` was the only thing in /usr/bin, which I assumed was pretty much exactly for this reason.
I think you're mixing two concepts: relative paths (which are allowed after #! but not very useful at all) and file lookup through $PATH (which is not done by the kernel, maybe it's some shell trickery).
> and file lookup through $PATH (which is not done by the kernel, maybe it's some shell trickery)
It's libc. Specifically, system(3) and APIs like execvp(3) will search $PATH for the file specified before passing that to execve(2) (which is the only kernel syscall; all the other exec*() stuff is in libc).
> /usr/bin/env and /bin/sh are part of the POSIX standard, this is why NixOS has those available.
Contrary to popular belief, those aren't in the POSIX standard.
The following are not in the POSIX standard, they are just widely implemented:
- "#!" line.
- /bin/sh as the location of a POSIX compliant shell, or any shell.
- /usr/bin/env as the location of an env program.
- The -S option to env.
I think you are using "not required by the POSIX standard" when you say "not in" which is not an accurate shorthand.
#! is certainly in the POSIX standard as the exact topic of "is /bin/sh always a POSIX" shell is a discussion point (it is not guaranteed since there were systems that existed at the time that had a non-POSIX shell there)
Are they in POSIX? I do not think they are. All of them is a convention from what I remember.
Shebang is a kernel feature, for example, and POSIX does define the sh shell language and utilities, but does not specify how executables are invoked by the kernel.
Similarly, POSIX only requires that sh exists somewhere in the PATH, and the /bin/sh convention comes from the traditional Unix and FHS (Filesystem Hierarchy Standard), but POSIX does not mandate filesystem layout.
You're definitely correct. "#!" is reserved (see Rationale C.2.1), but not required, though it's described as "ubiquitous" (see Rationale C.1.7). "/bin/sh" isn't required either, but arguably ubiquitous in that there's always some shell located there. The proper way to find the POSIX-conformant shell is with `command -v sh` (which is equivalent to using `getconf PATH` and then searching for sh), and POSIX counsels to discover the path and substitute it inline when installing scripts (see Application Usage in sh utility specification.)
IME /bin/sh is invariably sufficiently POSIX conformant to bootstrap into a POSIX shell (or your preferred shell), even on Solaris and AIX. And if you're willing to stick to simple scripts or rigorously test across systems, sufficient for most tasks. Outside Linux-based systems it's usually ksh88, ksh93, pdksh, or some derivative. OTOH, for those who are only familiar with bash that may not be particularly helpful.
I've had more trouble, including bugs, with other utilities, like sed, tr, paste, etc. For shell portability it's usually niche stuff like "$@" expansion with empty lists, for example how it interacts with nounset or IFS, independent of POSIX mode.
Linux is UNIX in the context of my question. On Linux the shebang is actually handled by the kernel. When you load a binary and attempt to execute it with a syscall, the kernel reads the first few bytes of the binary. If it is an ELF header, it executes the machine code as you would expect. If the first two bytes are "#!", then it interprets it as a shebang header and loads the specified binary to interpret it.
Again, this is kernel code. I admit I'm a bit confused as to why it didn't work on your system. This shouldn't be handled at the shell level.
It is possible for the shell to handle it. From zshall(1):
> If the program is a file beginning with ‘#!', the remainder of the first line specifies an interpreter for the program. The shell will execute the specified interpreter on operating systems that do not handle this executable format in the kernel.
I did a little digging and found that the `|| eno == ENOENT` was added quite a bit earlier[1] than the actual pathprog lookup[2]. While I could find the "issue discussion" for the pathprog change[3] I wasn't able to find it for the ENOENT addition, which was kind of interesting and frustrating--[4] is the `X-Seq` mentioned in the commit but that seems to be inconsistent or incorrect for the actual cross-reference, and nearby in time wasn't helpful either.
I think it makes it to calling open_exec but there's a test for BINPRM_FLAGS_PATH_INACCESSIBLE, which doesn't seem relevant since 'bash' isn't like '/dev/fd/<fd>/..', but does provoke an ENOENT.
env bash is all well and good for normies, but if you're already on NixOS did you know you can have nix-shell be your interpreter and back flip into any reproducible interpreted environment you like?
Or any other system with Nix installed. I use this at work to provide scripts with all their dependencies specified that work across any Linux distro & MacOS. First execution is slow since it has to fetch everything, but after that it's fast and just works.
Hit the link expecting to read about UTF-8 Byte Order Marks at the top of the file, so that the first few bytes aren't actually #! but 0xEF 0xBB 0xBF #! instead. Ran into this one just a few months ago when a coworker who uses Windows had checked a Bash script into the Git repo. His editor was configured to save files as "UTF-8 with BOM" and so we were getting errors that looked like "./doit.sh: line 1: #!/bin/bash: No such file or directory". Can you see the invisible BOMb in that line? It's there, I promise you.
That's not what the article was actually about, as it turned out. The surprise in the article was about relative paths for script shebang lines. Which was useful to learn about, of course, but I was actually surprised by the surprise.
UTF-8 doesn't have a BOM. UTF-16 does. UTF-32 does. "UTF-8 with BOM" is not a standards-compliant text format, it's a proprietary binary format that happens to have a bunch of embedded UTF-8. Just because you can run `strings` on a file & get a bunch of text out doesn't mean it's a text file!
This seems a bit pedantic, while you may be correct (I honestly don't know what standard this is referring to) the UTF-8 BOM is a thing that some tools do know about. Even then in the context of OP's question the BOM with UTF-8 isn't the specific problem but rather how the shebang interpreter reads the actual ASCII byte sequences so a UTF-16 with a BOM "text" file would also fail.
UTF-8 is a text format with no BOM. Just like ASCII doesn't support a BOM. The BOM is a UTF-16 or UTF-32 thing, so "UTF-8 with BOM" is a binary file that happens to contain some UTF-8 strings as well. Since it's not a text file, it makes sense that utilities expecting text files don't handle it.
I do this all the time for python virtualenvs. Instead of calling ‘#!/usr/bin/env python3’ I will call ‘#!venv/bin/python3’. This way I don’t have to worry about whether the environment was activated or not.
If you're okay with running the script only from the parent directory of the venv (I guess you have things set up so that's the project root), fine.
You never have to "worry about whether the environment was activated", unless your code depends on those environment variables (in which case your shebang trick won't help you). Just specify the path to the venv's Python executable.
You aren't really intended to put your own scripts in the venv's bin/ directly, although of course you can. An installer will create them for you, from the entry points defined in pyproject.toml. (This is one of the few useful things that an "editable install" can accomplish; I'm generally fairly negative on that model, however.)
If you have something installed in a venv and want to run it regardless of CWD, you can symlink the wrapper script from somewhere on your PATH. (That's what Pipx does.)
I use venv's to keep library installs out of the global space. My python scripts won't run without those libraries, so running from the project root is pretty much required. I don't use the variables set by activating the venv, so that's not a real concern.
I've done this for years to keep my individual projects separate and not changing activations when switching directories. I also make sure to only call `venv/bin/pip` or `venv/bin/python3` when I'm directly calling pip or python. So, yes -- you have to be in the root project directory for this to work, but for me, that's a useful tradeoff. Even when running code from within a docker container, I still use this method, so I make sure that I'm executing from the proper work directory.
If I think that I need to run a program (without arguments), I'll have a short shell wrapper that is essentially:
#!/bin/bash
cd $(dirname $0)
venv/bin/python3 myscript.py
As far as running a program that's managed by venv/pip, symlinks are essentially what I do. I'll create a new venv for each program, install it in that venv, and then symlink from venv/bin/program to $HOME/.local/bin/. It works very well when you're installing a pip managed program.
This is terrible. I hope you're not adding a whole venv to version control.
What if the user doesn't have a venv created? What if they created it in a different directory? What if they created a second venv and want to use that instead? What if the user uses `.venv` instead of `venv`?
These are programs that are largely meant to have a run.sh or install.sh script run before the main script. If the venv doesn’t exist, the it is created there and requirements installed.
The main point is that I’m trading away some flexibility to keep my ENV clean. When I submit jobs on HPC clusters, keeping my environment clean tends to make things easier to troubleshoot.
If I’m switching between different programs or commonly piping data between two different programs with their own venvs, it can be easier to just run the associated python binary directly, rather than have to manage different venv environment activations.
You can have your cake (use multiple venvs) and eat it (flexibility) too.
`source venv/bin/activate` from your .sh files will cause `/usr/bin/env python3` to use the python3 located in your venv. Switching between venvs is easy too. just call `deactivate` when one venv is activated. It drops you out of the venv. You can then cleanly `source venv2/bin/activate`.
It feels like you're telling me that I'm holding my phone wrong.
I don't like sourcing things into my environment. I've worked this way for years. I think the idea of 'activating' and 'deactivating' an environment is an anti-pattern. But I also work on HPC clusters where all of the configuration about paths is handled by the environment. Because of this, I've learned the hard way that for my workflows, it's far too easy to have the wrong environment loaded with venvs and modules that it's often better to keep things explicitly defined. I don't like magic, so I explicitly state which venv my code (or occasionally other people's code) is loading from.
I sometimes will have to run multiple programs (that have their own venvs) and pipe data between them. If I have to source and deactivate different venvs for each tool, it just doesn't work right.
I think that's part of the power of virtualenv as a tool -- it's flexible in how it works. I can still use my explicit workflows with the same tooling as everyone else, and you can source your environments and keep happily coding along. For me, that's why I keep using them...
> Although this is probably the easiest way to implement '#!' inside the kernel, I'm a little bit surprised that it survived in Linux (in a completely independent implementation) and in OpenBSD (where the security people might have had a double-take at some point). But given Hyrum's Law there are probably people out there who are depending on this behavior so we're now stuck with it.
I don't see what there would be to gain in disallowing the program path on the shebang line to be relative. The person that wrote the shebang can also write relative paths in some other part of the file.
Or, like, if you aren't reading and caring about what the interpreter is--as that's the only time this can burn you: it isn't doing a PATH lookup, so you can't walk into this one on accident--then it could literally be something like /bin/rm on some key file. This entire article is based on an assumption that this is somehow so obviously bad that there doesn't even need to be an explanation or defense of any kind of that idea.
> This entire article is based on an assumption that this is somehow so obviously bad that there doesn't even need to be an explanation or defense of any kind of that idea.
I'm not reading it like that. The tone is just one of surprise, since this isn't something that one typically sees. Since it's obscure, it leads one to wonder if it can be bad, and I don't see how it could be.
I think it survived in the independent Linux because it's the simple and obvious way to do things, and it doesn't lead to any exceptional power of misuse one didn't already have with writing the rest of the file.
The authors previous article about "...(not) using #!/usr/bin/env whatever" doesn't sit well with me.
"The only reason to start your script with '#!/usr/bin/env <whatever>' is if you expect your script to run on a system where Bash or whatever else isn't where you expect (or when it has to run on systems that have '<whatever>' in different places, which is probably most common for third party packages)."
His very first point is how you should only use it don't know where to expect bash binary, when I feel like, while it's probably safe in most nix os', assuming it limits future enhancements by requiring that binary be in place. However unlikely it would need to or someone would want to move it.
I've encountered systems that only have bash in /bin/bash, or in /usr/bin/bash, and it's a hell of a pain to have to fix every script when using different distros (I think it must've been an old Fedora and Ubuntu?).
Nowadays, most distros are moving towards having /bin be a symlink to /usr/bin, so it's mattering less and less, but I see no reason not to just do /usr/bin/env which is supposed to be on the same place on every distro.
I use `#!/usr/bin/env bash` because the /bin/bash on macOS is usually stupidly ancient, and I install a recent version of bash to ~/.local/bin/bash
Is it bad? Well it's less secure. But if you're worried about `/usr/bin/env` calling a malicious program then you need to call out the path for every executable in the script and there's a hell of a lot more other things to worry about too.
It's the same for `#!/usr/bin/env python3` in python scripts. Python3 itself might be ancient at system install, but you might need to be using a venv. So /usr/bin/env python3 works correctly while /usr/bin/python3 works incorrectly.
Never thought of it this way; isn’t it always safe to assume env is in PATH?
Maybe `#! env <shell>` could be considered a DSL for hashbangs. My reasoning is that `/usr/bin/env` is the thing that seems to be hard-coded to a system path, in most cases.
I wonder what the reason was for having the kernel handle this, instead of the shell? To allow programs besides the shell to execute interpreted scripts as if they were actual binaries?
This is of course in stark contrast to dynamic linking, which is performed by a userspace program instead of the kernel, and much like the #!, this "interpreter"'s path is also hardcoded in dynamically linked binaries.
In a call `execve("/my/script", ...)` of course the kernel has to figure out how to run it, there is no shell involved.
As for scripts vs elf executables, there's not much of a difference between the shebang line and PT_INTERP, just that parsing shebangs lines is simpler.
Remember how slow early machines were. An unnecessary additional 'exec' would have seemed very wasteful, especially as most of the initial uses would have been shell scripts, so you would be execing the shell twice.
For personal use, I used to write #!perl to avoid the hassle of keeping track of the location of the binary across differing systems (although now I wonder if this only worked in the shell I ran under WinNT4—it’s been ages since I’ve done that sort of thing and these days I tend not to use #! at all to run scripts.)
Historically FreeBSD used to split the #! argument list fully which meant you could put oneline scripts in there. At some point too many ports had to struggle with Linuxisms that this was changed to only split it into command and first argument. The old behaviour can be accessed with a flag in the env wrapper.
There is no security issue here. The file with the '#!' needs to be executable, and at that point it doesn't matter what it invokes because you made it executable. It could have shellcode in it or it could call python3 which can also execute shellcode. Or more likely, it would just be a malware binary which you deliberately gave permissions to and executed.
It's a vulnerability via pathing, not a worry that the shebang script could be malicious.
Someone may have dropped a malicious executable somewhere in the user's path that the shebang calls. The someone shouldn't be able to do that, but "shouldn't" isn't enough for security.
Or maybe the relatively pathed executable has unexpected interactions with the shebanged script, compared to what the script author expected.
This example works on many platforms that have a shell compatible with Bourne shell:
#!/usr/bin/perl
eval 'exec /usr/bin/perl -wS $0 ${1+"$@"}'
if 0; # ^ Run only under a shell
The system ignores the first line and feeds the program to /bin/sh, which proceeds to try to execute the Perl program as a shell script. The shell executes the second line as a normal shell command, and thus starts up the Perl interpreter. On some systems $0 doesn't always contain the full pathname, so the "-S" tells Perl to search for the program if necessary. After Perl locates the program, it parses the lines and ignores them because the check 'if 0' is never true. If the program will be interpreted by csh, you will need to replace ${1+"$@"} with $*, even though that doesn't understand embedded spaces (and such) in the argument list. To start up sh rather than csh, some systems may have to replace the #! line with a line containing just a colon, which will be politely ignored by Perl. Other systems can't control that, and need a totally devious construct that will work under any of csh, sh, or Perl, such as the following:
eval '(exit $?0)' && eval 'exec perl -wS $0 ${1+"$@"}'
& eval 'exec /usr/bin/perl -wS $0 $argv:q'
if 0; # ^ Run only under a shell
The thing that surprised me was that you can't write an interpreter in an interpreted language, at least not in obsd. It is possible if you jump through a few hoops but you can't directly call it.
An example: if you made a language in python /bin/my_lang: #does nothing but pretend it does
my_script: Probably for the best, but I was a bit sad that my recursive interpreter scheme was not going to work.Update: looks like linux does allow nested interpreters, good for them.
https://www.in-ulm.de/~mascheck/various/shebang/#interpreter...
really that whole document is a delightful read.
Worked for me, but the way you described it has issues:
1. You chmod my_script twice.
2. Did you chmod u+x /bin/my_lang too? Since you put it in /bin, are you sure the owner isn't root?, in which case your user wouldn't have execute permission. Try +x instead of u+x.
3. Do you have python in that path? Try `/usr/bin/env python` instead.
4. In case you expected otherwise, my_script wouldn't be passed through stdin. It's just provided as an argument to my_lang.
I am on openbsd. which does not allow it, it looks like nested interpreters are s supported on linux. So my loss there.
Wait... your interpreter reads from stdin. Shouldn't it read its first arg, instead?
IIRC interpreters can also read their programs from stdin. Try piping a script to bash and see if it works.
(Actually, this is how the `curl install.sh | bash` anti pattern works. )
Independently of what the interpreter can do, it's #! that doesn't pass the file to it.
I think that was what I was trying to figure out, how the program was passed. but OpenBSD does not do nested interpreters, it looks like if I had tried Linux it would have worked.
Huh. I wish I had known this before.
NixOS is annoying because everything is weird and symlinked and so I find myself fairly frequently making the mistake of writing `#!/bin/bash`, only to be told it can't find it, and I have to replace the path with `/run/current-system/sw/bin/bash`.
Or at least I thought I did; apparently I can just have done `#!bash`. I just tested this, and it worked fine. You learn something new every day I guess.
Anything other than ”#!/usr/bin/env bash” is doomed to fail at some time.
If that fails, I assume the user knows enough about their environment to fix it.
And is this shebang guaranteed to work always? Why isn't it more common?
Because /bin is the standard location for bash. The only one that breaks that expectation is NixOS (and maybe GuixSD?), apparently. I'm surprised they didn't symlink /bin or put a stub. Last time I tried NixOS was like 10 years ago. I thought there was a /bin/bash, but maybe it was just a /bin/sh?
Other interpreters like python, ruby, etc. have more likelyhood of being used with "virtual environments", so it's more common to use /usr/bin/env with them.
/bin is the "standard" location for bash on a subset of Linux distributions and basically no other Unix...
So it's not really a standard.
/bin/sh is a much more common convention but once again, not a standard.
There really isn't a truly portable shebang, but the same can be said about executables themselves. As part of the build or install step of whatever thing you're making, you should really be looking these up and changing them.
What's more, bash isn't a standard shell.
Sorry, I should probably think more widely, but I was just considering Linux distros.
> /bin is the "standard" location for bash on a subset of Linux distributions
Considering "location" such that it includes /bin symlinks, that would be nearly all distros, I would think...
> What's more, bash isn't a standard shell.
De facto and specifically among Linux distros, it is. It's probably an underestimate that 95% of all Linux distro installations have it preinstalled.
It's only really NixOS as far as I know that doesn't ever put bash in /bin/bash (as far as Linux distributions go). But, on the other hand, there are quite a few distros (or at least flavours of distros) which don't ship bash by default (alpine, minimal versions of most distros, and embedded-Linux focused stuff if you count it). I imagine the most common "installation" of Linux is userspace in a container (yeah I know there's no kernel there, but nobody who talks about "Linux" broadly speaking specifically cares about the kernel) and a good chunk of those will be minimal with no bash.
Bash has to be explicitly installed on OpenBSD, FreeBSD, NetBSD (I think, haven't used it in a while) and probably a bunch of others. And in all of those cases (that I know of) it doesn't end up in /bin/bash once installed.
The default bash shipped on macs is so abhorrently ancient that it would be strictly better if it didn't exist because it would reduce the number of people who think bash scripts I write are broken (they're not broken, they just inevitably depend on some bash 4+ feature). Moreover, hardcoding /bin/bash as your shebang in this case will prevent anyone from remediating this problem by installing a non-ancient bash because the old one in /bin/bash will still get used.
> /bin is the "standard" location for bash on a subset of Linux distributions and basically no other Unix...
You’re forgetting macOS. It has been using /bin/bash forever.
Keep in mind that the bash you get on MacOS is bash 3.2 released in 2006 so relying on it for portability might not be a good idea.
Pretty much. I will continue using "#!/usr/bin/env <language>".
Will not wok on OpenBSD where the shell that comes with the system is ksh at /bin/ksh and /bin/sh and if you want bash it is a third party package and correspondingly gets installed as /usr/local/bin/bash
It does get awkward, especially when porting. all your third party libraries and includes are in /usr/local/lib /usr/local/include but at least it is better than freebsd which also insists on putting all third party configs under /usr/local/etc/
They do symlink /bin/sh to be fair, and that's very often good enough for a lot of scripts. That's what I usually do if I don't need anything bash offers.
Thing is, a few years ago when Debian changed its default sh from bash to ... either ash or dash, I forget which, I got into the habit of always writing `#!/bin/bash` at the top of my scripts, in case I didn't realize that something I was using was a bashism not found in classic /bin/sh. So if I used Nix (I don't, since for my particular use cases the juice isn't worth the squeeze), I would get seriously messed up by that.
/bin/sh is part of the POSIX standard -- even NixOS puts a symlink there.
It's guaranteed to work provided that Bash is in the path.
It's very common for Python. Less so for Bash for two reasons: because the person who writes the script references /bin/sh instead (which is required to be there) even when they are writing bash-isms, or because the person who writes the script assumes that Bash is universally available as /bin/bash.
It’s quite common, although I probably see it used more frequently to invoke other (non-shell) scripting languages.
> Anything other than ”#!/usr/bin/env bash” is doomed to fail at some time.
if you have /usr/bin/env
/usr/bin/env (and /bin/sh) are part of POSIX, that is why the above shebang is the recommended way to start a shell.
/bin/sh is NOT required by POSIX, they explicitly warn that it may not exist[1].
> Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH , ensuring that the returned pathname is an absolute pathname and not a shell built-in.
[1] https://pubs.opengroup.org/onlinepubs/9799919799/
I don't know if this is still a thing, but I distinctly remember when playing around with NixOS years ago that `env` was the only thing in /usr/bin, which I assumed was pretty much exactly for this reason.
> apparently I can just have done `#!bash`
I think you're mixing two concepts: relative paths (which are allowed after #! but not very useful at all) and file lookup through $PATH (which is not done by the kernel, maybe it's some shell trickery).
> and file lookup through $PATH (which is not done by the kernel, maybe it's some shell trickery)
It's libc. Specifically, system(3) and APIs like execvp(3) will search $PATH for the file specified before passing that to execve(2) (which is the only kernel syscall; all the other exec*() stuff is in libc).
You can use `#!/usr/bin/env bash` on NixOS
I didn't know that actually. I'll start using that from this point forward.
/usr/bin/env and /bin/sh are part of the POSIX standard, this is why NixOS has those available.
> /usr/bin/env and /bin/sh are part of the POSIX standard, this is why NixOS has those available.
Contrary to popular belief, those aren't in the POSIX standard.
The following are not in the POSIX standard, they are just widely implemented:
I think you are using "not required by the POSIX standard" when you say "not in" which is not an accurate shorthand.
#! is certainly in the POSIX standard as the exact topic of "is /bin/sh always a POSIX" shell is a discussion point (it is not guaranteed since there were systems that existed at the time that had a non-POSIX shell there)
Are they in POSIX? I do not think they are. All of them is a convention from what I remember.
Shebang is a kernel feature, for example, and POSIX does define the sh shell language and utilities, but does not specify how executables are invoked by the kernel.
Similarly, POSIX only requires that sh exists somewhere in the PATH, and the /bin/sh convention comes from the traditional Unix and FHS (Filesystem Hierarchy Standard), but POSIX does not mandate filesystem layout.
... and so on.
Correct me if I am wrong, perhaps with citations?
You're definitely correct. "#!" is reserved (see Rationale C.2.1), but not required, though it's described as "ubiquitous" (see Rationale C.1.7). "/bin/sh" isn't required either, but arguably ubiquitous in that there's always some shell located there. The proper way to find the POSIX-conformant shell is with `command -v sh` (which is equivalent to using `getconf PATH` and then searching for sh), and POSIX counsels to discover the path and substitute it inline when installing scripts (see Application Usage in sh utility specification.)
IME /bin/sh is invariably sufficiently POSIX conformant to bootstrap into a POSIX shell (or your preferred shell), even on Solaris and AIX. And if you're willing to stick to simple scripts or rigorously test across systems, sufficient for most tasks. Outside Linux-based systems it's usually ksh88, ksh93, pdksh, or some derivative. OTOH, for those who are only familiar with bash that may not be particularly helpful.
I've had more trouble, including bugs, with other utilities, like sed, tr, paste, etc. For shell portability it's usually niche stuff like "$@" expansion with empty lists, for example how it interacts with nounset or IFS, independent of POSIX mode.
It is good practice to be using it everywhere.
Assuming bash is in $PATH. Which it often is.
Seems like it only works in zsh, not bash or fish
Is this UNIX?
This is NixOS, so no, it's Linux. I guess I just hoped it would work on Linux as well.
Linux is UNIX in the context of my question. On Linux the shebang is actually handled by the kernel. When you load a binary and attempt to execute it with a syscall, the kernel reads the first few bytes of the binary. If it is an ELF header, it executes the machine code as you would expect. If the first two bytes are "#!", then it interprets it as a shebang header and loads the specified binary to interpret it.
Again, this is kernel code. I admit I'm a bit confused as to why it didn't work on your system. This shouldn't be handled at the shell level.
The kernel interprets the shebang line, not the shell.
It is possible for the shell to handle it. From zshall(1):
> If the program is a file beginning with ‘#!', the remainder of the first line specifies an interpreter for the program. The shell will execute the specified interpreter on operating systems that do not handle this executable format in the kernel.
Taking a quick look at the source in Src/exec.c:
I guess at some point someone added that `|| eno == ENOENT` and the docs weren't updated.I did a little digging and found that the `|| eno == ENOENT` was added quite a bit earlier[1] than the actual pathprog lookup[2]. While I could find the "issue discussion" for the pathprog change[3] I wasn't able to find it for the ENOENT addition, which was kind of interesting and frustrating--[4] is the `X-Seq` mentioned in the commit but that seems to be inconsistent or incorrect for the actual cross-reference, and nearby in time wasn't helpful either.
[1] https://sourceforge.net/p/zsh/code/ci/29ed6c7e3ab32da20f528a...
[2] https://sourceforge.net/p/zsh/code/ci/29ed6c7e3ab32da20f528a...
[3] https://www.zsh.org/mla/workers/2010/msg00522.html
[4] https://www.zsh.org/mla/workers/2000/msg01168.html
I'm not sure the reason then, but they're definitely right; it works fine with zsh, doesn't work with bash. I wrote a test script to try it myself.
I don't have fish installed and can't be bothered to go that far, but I suspect they're right about that as well.
It is strange, cursory digging for an explanation was a little more complex than I bargained for...
https://github.com/torvalds/linux/blob/v6.17/fs/binfmt_scrip...
I think it makes it to calling open_exec but there's a test for BINPRM_FLAGS_PATH_INACCESSIBLE, which doesn't seem relevant since 'bash' isn't like '/dev/fd/<fd>/..', but does provoke an ENOENT.
https://github.com/torvalds/linux/blob/v6.17/fs/exec.c#L1445
Maybe someone else can explain it, I'd enjoy the details, and ran out of steam.
env bash is all well and good for normies, but if you're already on NixOS did you know you can have nix-shell be your interpreter and back flip into any reproducible interpreted environment you like?
https://nixos.wiki/wiki/Nix-shell_shebang
Or any other system with Nix installed. I use this at work to provide scripts with all their dependencies specified that work across any Linux distro & MacOS. First execution is slow since it has to fetch everything, but after that it's fast and just works.
`#!/usr/bin/env bash` is the most portable form for executing it from $PATH
I raise you a https://www.felesatra.moe/blog/2021/07/03/portable-bash-sheb...
Pedantic, but "#!" and "portable" don't belong in the same sentence
Is this meaningfully more portable than #!bash though?
In a sibling thread someone pointed out that #!bash doesn't actually work if you're calling it from bash, and appears to only work with zsh.
I just tried it and they were absolutely right, so `#!/usr/bin/env bash` is definitely more portable in that it consistently works.
This mechanism doesn't do a PATH lookup: #!bash would only work if bash was located in your current working directory.
Hit the link expecting to read about UTF-8 Byte Order Marks at the top of the file, so that the first few bytes aren't actually #! but 0xEF 0xBB 0xBF #! instead. Ran into this one just a few months ago when a coworker who uses Windows had checked a Bash script into the Git repo. His editor was configured to save files as "UTF-8 with BOM" and so we were getting errors that looked like "./doit.sh: line 1: #!/bin/bash: No such file or directory". Can you see the invisible BOMb in that line? It's there, I promise you.
That's not what the article was actually about, as it turned out. The surprise in the article was about relative paths for script shebang lines. Which was useful to learn about, of course, but I was actually surprised by the surprise.
UTF-8 doesn't have a BOM. UTF-16 does. UTF-32 does. "UTF-8 with BOM" is not a standards-compliant text format, it's a proprietary binary format that happens to have a bunch of embedded UTF-8. Just because you can run `strings` on a file & get a bunch of text out doesn't mean it's a text file!
This seems a bit pedantic, while you may be correct (I honestly don't know what standard this is referring to) the UTF-8 BOM is a thing that some tools do know about. Even then in the context of OP's question the BOM with UTF-8 isn't the specific problem but rather how the shebang interpreter reads the actual ASCII byte sequences so a UTF-16 with a BOM "text" file would also fail.
tbh it is lame for any program reading a text file to not support BOM. It's just one if.
There isn't really any one "text file" though, the kernel looks for the first two bytes to match what "#!" corresponds to in ASCII.
https://www.youtube.com/watch?v=J8nblo6BawU is some great watching on how "Plain text isn't that simple"
UTF-8 is a text format with no BOM. Just like ASCII doesn't support a BOM. The BOM is a UTF-16 or UTF-32 thing, so "UTF-8 with BOM" is a binary file that happens to contain some UTF-8 strings as well. Since it's not a text file, it makes sense that utilities expecting text files don't handle it.
Eh? A utf8 file starting with ZERO WIDTH NO-BREAK SPACE is not a text file? How do you figure that?
I do this all the time for python virtualenvs. Instead of calling ‘#!/usr/bin/env python3’ I will call ‘#!venv/bin/python3’. This way I don’t have to worry about whether the environment was activated or not.
If you're okay with running the script only from the parent directory of the venv (I guess you have things set up so that's the project root), fine.
You never have to "worry about whether the environment was activated", unless your code depends on those environment variables (in which case your shebang trick won't help you). Just specify the path to the venv's Python executable.
You aren't really intended to put your own scripts in the venv's bin/ directly, although of course you can. An installer will create them for you, from the entry points defined in pyproject.toml. (This is one of the few useful things that an "editable install" can accomplish; I'm generally fairly negative on that model, however.)
If you have something installed in a venv and want to run it regardless of CWD, you can symlink the wrapper script from somewhere on your PATH. (That's what Pipx does.)
I use venv's to keep library installs out of the global space. My python scripts won't run without those libraries, so running from the project root is pretty much required. I don't use the variables set by activating the venv, so that's not a real concern.
I've done this for years to keep my individual projects separate and not changing activations when switching directories. I also make sure to only call `venv/bin/pip` or `venv/bin/python3` when I'm directly calling pip or python. So, yes -- you have to be in the root project directory for this to work, but for me, that's a useful tradeoff. Even when running code from within a docker container, I still use this method, so I make sure that I'm executing from the proper work directory.
If I think that I need to run a program (without arguments), I'll have a short shell wrapper that is essentially:
As far as running a program that's managed by venv/pip, symlinks are essentially what I do. I'll create a new venv for each program, install it in that venv, and then symlink from venv/bin/program to $HOME/.local/bin/. It works very well when you're installing a pip managed program.I also use the trick to insert new lookup paths.
This is terrible. I hope you're not adding a whole venv to version control.
What if the user doesn't have a venv created? What if they created it in a different directory? What if they created a second venv and want to use that instead? What if the user uses `.venv` instead of `venv`?
`#!/usr/bin/env python3` solves most of that.
No!
These are programs that are largely meant to have a run.sh or install.sh script run before the main script. If the venv doesn’t exist, the it is created there and requirements installed.
The main point is that I’m trading away some flexibility to keep my ENV clean. When I submit jobs on HPC clusters, keeping my environment clean tends to make things easier to troubleshoot.
If I’m switching between different programs or commonly piping data between two different programs with their own venvs, it can be easier to just run the associated python binary directly, rather than have to manage different venv environment activations.
You can have your cake (use multiple venvs) and eat it (flexibility) too.
`source venv/bin/activate` from your .sh files will cause `/usr/bin/env python3` to use the python3 located in your venv. Switching between venvs is easy too. just call `deactivate` when one venv is activated. It drops you out of the venv. You can then cleanly `source venv2/bin/activate`.
It feels like you're telling me that I'm holding my phone wrong.
I don't like sourcing things into my environment. I've worked this way for years. I think the idea of 'activating' and 'deactivating' an environment is an anti-pattern. But I also work on HPC clusters where all of the configuration about paths is handled by the environment. Because of this, I've learned the hard way that for my workflows, it's far too easy to have the wrong environment loaded with venvs and modules that it's often better to keep things explicitly defined. I don't like magic, so I explicitly state which venv my code (or occasionally other people's code) is loading from.
I sometimes will have to run multiple programs (that have their own venvs) and pipe data between them. If I have to source and deactivate different venvs for each tool, it just doesn't work right.
I think that's part of the power of virtualenv as a tool -- it's flexible in how it works. I can still use my explicit workflows with the same tooling as everyone else, and you can source your environments and keep happily coding along. For me, that's why I keep using them...
> Although this is probably the easiest way to implement '#!' inside the kernel, I'm a little bit surprised that it survived in Linux (in a completely independent implementation) and in OpenBSD (where the security people might have had a double-take at some point). But given Hyrum's Law there are probably people out there who are depending on this behavior so we're now stuck with it.
I don't see what there would be to gain in disallowing the program path on the shebang line to be relative. The person that wrote the shebang can also write relative paths in some other part of the file.
Or, like, if you aren't reading and caring about what the interpreter is--as that's the only time this can burn you: it isn't doing a PATH lookup, so you can't walk into this one on accident--then it could literally be something like /bin/rm on some key file. This entire article is based on an assumption that this is somehow so obviously bad that there doesn't even need to be an explanation or defense of any kind of that idea.
> This entire article is based on an assumption that this is somehow so obviously bad that there doesn't even need to be an explanation or defense of any kind of that idea.
I'm not reading it like that. The tone is just one of surprise, since this isn't something that one typically sees. Since it's obscure, it leads one to wonder if it can be bad, and I don't see how it could be.
I think it survived in the independent Linux because it's the simple and obvious way to do things, and it doesn't lead to any exceptional power of misuse one didn't already have with writing the rest of the file.
Right: I agree with you. I'm saying the article is making an unfounded assumption and am providing more reasoning for why you are correct.
What's suprised me is that there is a line length limit on #! for 256 bytes.
You can have spacing after #! for some reason?
POSIX does not mandate -S, which means any script that uses it will only work on freebsd/linux
The authors previous article about "...(not) using #!/usr/bin/env whatever" doesn't sit well with me.
"The only reason to start your script with '#!/usr/bin/env <whatever>' is if you expect your script to run on a system where Bash or whatever else isn't where you expect (or when it has to run on systems that have '<whatever>' in different places, which is probably most common for third party packages)."
His very first point is how you should only use it don't know where to expect bash binary, when I feel like, while it's probably safe in most nix os', assuming it limits future enhancements by requiring that binary be in place. However unlikely it would need to or someone would want to move it.
I've encountered systems that only have bash in /bin/bash, or in /usr/bin/bash, and it's a hell of a pain to have to fix every script when using different distros (I think it must've been an old Fedora and Ubuntu?).
Nowadays, most distros are moving towards having /bin be a symlink to /usr/bin, so it's mattering less and less, but I see no reason not to just do /usr/bin/env which is supposed to be on the same place on every distro.
I use `#!/usr/bin/env bash` because the /bin/bash on macOS is usually stupidly ancient, and I install a recent version of bash to ~/.local/bin/bash
Is it bad? Well it's less secure. But if you're worried about `/usr/bin/env` calling a malicious program then you need to call out the path for every executable in the script and there's a hell of a lot more other things to worry about too.
It's the same for `#!/usr/bin/env python3` in python scripts. Python3 itself might be ancient at system install, but you might need to be using a venv. So /usr/bin/env python3 works correctly while /usr/bin/python3 works incorrectly.
So is it bad? No.
Never thought of it this way; isn’t it always safe to assume env is in PATH?
Maybe `#! env <shell>` could be considered a DSL for hashbangs. My reasoning is that `/usr/bin/env` is the thing that seems to be hard-coded to a system path, in most cases.
I think POSIX requires env but doesn't mandate a location for it.
Some environments (like containers) have no, or extremely incomplete PATHs.
NixOS only has /usr/bin/env and /bin/sh.
I wonder what the reason was for having the kernel handle this, instead of the shell? To allow programs besides the shell to execute interpreted scripts as if they were actual binaries?
This is of course in stark contrast to dynamic linking, which is performed by a userspace program instead of the kernel, and much like the #!, this "interpreter"'s path is also hardcoded in dynamically linked binaries.
In a call `execve("/my/script", ...)` of course the kernel has to figure out how to run it, there is no shell involved.
As for scripts vs elf executables, there's not much of a difference between the shebang line and PT_INTERP, just that parsing shebangs lines is simpler.
Remember how slow early machines were. An unnecessary additional 'exec' would have seemed very wasteful, especially as most of the initial uses would have been shell scripts, so you would be execing the shell twice.
For personal use, I used to write #!perl to avoid the hassle of keeping track of the location of the binary across differing systems (although now I wonder if this only worked in the shell I ran under WinNT4—it’s been ages since I’ve done that sort of thing and these days I tend not to use #! at all to run scripts.)
> You're using a suspiciously old browser
Does the author not like Firefox or what?
I'm on FF 145 and don't see this message.
There's no security issue here. Certainly the OP hasn't explained why there is one.
There could be. Say you have '#!sh', someone could have put a broken 'sh' somewhere, that could be the first one the '#!' sees.
more/older https://www.in-ulm.de/~mascheck/various/shebang/
Historically FreeBSD used to split the #! argument list fully which meant you could put oneline scripts in there. At some point too many ports had to struggle with Linuxisms that this was changed to only split it into command and first argument. The old behaviour can be accessed with a flag in the env wrapper.
There is no security issue here. The file with the '#!' needs to be executable, and at that point it doesn't matter what it invokes because you made it executable. It could have shellcode in it or it could call python3 which can also execute shellcode. Or more likely, it would just be a malware binary which you deliberately gave permissions to and executed.
It's a vulnerability via pathing, not a worry that the shebang script could be malicious.
Someone may have dropped a malicious executable somewhere in the user's path that the shebang calls. The someone shouldn't be able to do that, but "shouldn't" isn't enough for security.
Or maybe the relatively pathed executable has unexpected interactions with the shebanged script, compared to what the script author expected.
Etc.
This insight came from perl5 btw.
See also the perlrun documentation[0].
This example works on many platforms that have a shell compatible with Bourne shell:
The system ignores the first line and feeds the program to /bin/sh, which proceeds to try to execute the Perl program as a shell script. The shell executes the second line as a normal shell command, and thus starts up the Perl interpreter. On some systems $0 doesn't always contain the full pathname, so the "-S" tells Perl to search for the program if necessary. After Perl locates the program, it parses the lines and ignores them because the check 'if 0' is never true. If the program will be interpreted by csh, you will need to replace ${1+"$@"} with $*, even though that doesn't understand embedded spaces (and such) in the argument list. To start up sh rather than csh, some systems may have to replace the #! line with a line containing just a colon, which will be politely ignored by Perl. Other systems can't control that, and need a totally devious construct that will work under any of csh, sh, or Perl, such as the following: [0]: https://perldoc.perl.org/perlrun#-S