This will be the first of a series of posts about the fundamentals of command-line shells in Unix-like systems: macOS, Linux, and Windows Subsystem for Linux (WSL). Also called “the terminal”, “the command line”, or simply a “shell”, the command-line shell is the primary tool for interacting with and an indispensable tool for programming. This is not a tutorial for how to get started using shells. There’s plenty of resources online that are better than what I can write. Rather, it’s meant to explain why shells typically behave they way they do, rather than simply how to use it. This knowledge can allow you to figure out your solution when you hit a problem with a script or command, without having to hope that someone else figured it out on StackOverflow.

The intended audience are people who some familiarity with shells and commonly have to use the command-line daily for programming work. Many of these ideas I didn’t learn myself until many years into my career, and once I did, I had the mental equivalent of getting the perfect piece to clear 4 rows in Tetris. The hundreds of commands, convoluted bash scripts, and one-liner incantations that I’d been copy-pasting for years finally made (some) sense!

I will be focusing on how so-called “POSIX-conformant (a standard outlining basic required functionality) shells work. POSIX-conformant shells includes bash, ash, dash, ksh, and zsh¹, among others. Each flavor of shell has additional features that are not discussed here, but all share the basic functionality.²

The goal of this post is to explain how a shell interprets the words and characters you type into it.

Strings all the way down

The only data structure that exists for shells are strings. Program invocations (“commands”), arguments, flags, return values—pretty much everything you type into the command-line is a string, though you might don’t realize it. There’s a few syntactic structures and reserved words including |, (), ;, &&, ||, >,if, else, fi, while, for, do, done. These will not be recognized as a string if typed outside a pair of quotes. Everything else is a string, including numbers and names of programs.

In other programming languages and environments, we’re used to strings being wrapped up in double- or single-quotes to indicate explicitly that they’re strings. Otherwise, it is a “bare word” that may refer to a variable or function. In the shell, we have the option to wrap things up in either double or single quotes, but it’s not necessary to create a string explicitly. Every sequence of characters in a shell that’s not reserved is a string or series of strings. By default, sequences of characters are separated into strings by whitespace. For instance,

echo everything is a string

is functionally the same as

"echo" "everything" "is" "a" "string"

If you’ve used shells for a while, this might seem strange to you. Copy it to a shell and see the result for yourself! For a long time, my mental model for understanding shell had been that echo or any other program name was similar to invoking a function in a programming language. echo had to be a bare word because it had special significance. Turns out, it’s just a lousy string like all others—only special because it’s first in the command prompt!

A future post will discuss how the shell knows which program to execute for this first string.

Variables substitute strings

Variables in shells assign a variable name to a string value. They persist for the scope of the variable (commonly for the life of the shell program) and can be used anywhere a string is. That is, you can use a variable to substitute for everything except reserved words and symbols.

The syntax for variables is simply alphanumeric_name_of_variable="string value". Variables can be referenced in a few ways, but we’ll simply use the most common dollar-sign syntax: $variable_name.

Traditionally, in shell scripts variable names are YELLED IN ALL CAPS. This is simply a convention, not a requirement you need to follow blindly. The all-cap convention may help distinguish variables in a shell script, but depending on your preferences and syntax-highlighting features in your text editor, you are welcome to use quieter tones in your variable names.

SO_LOUD="why are you yelling?!?"
echo $SO_LOUD # why are you yelling?!?

reasonable="much better" 
echo $reasonable # much better

Variables can be commands

If variables are strings and everything is a string in a shell, can we use variables to substitute for programs?

my_echo="echo"
$my_echo hello world # hello world

Indeed, we can! This neat trick is useful when you want to swap out the program being executed based on certain conditions. For example, macOS uses the BSD version of the program sed, which has different flags and options than the GNU sed program in Linux. Most frustratingly, the BSD version of sed lacks the -i option to allow you to edit files in place with find-and-replace filters. Users of macOS can install the GNU version of sed but it is typically named gsed. To account for the difference in names, we can use a variable to hold the appropriate variable name:

sed="sed"

# uname -s reports the name of the operating system.
# For macOS, this name is Darwin
if [[ $(uname -s) == Darwin ]]; then
	sed="gsed"
fi

$sed -i /find/replacement/g file_name

Double-quotes vs single-quotes

One of the initially confusing aspects of shells is when to use double-quotes and single-quotes, if at all. Both double- and single-quotes allow you to delineate strings which include whitespace. By default, shells split sequences of characters into separate strings between whitespace. If you want the whitespace to be considered part of the string, use quotes. For example,

# count_characters is a made-up program that counts the number
# of characters for each argument (string) entered and prints
# the count for each argument
count_characters hello world separate words
# hello: 5
# world: 5
# separate: 8
# words: 5
count_characters "hello world" separate
# hello world: 11
# separate: 8
# words: 5

The difference between double- and single-quotes is that double-quotes allow for interpolation of variables and escape characters, such as newline (\n) and escaping a double-quote (\")

name="Laska"
echo "\"Hello\", said $name.\n I turned and smiled."
# "Hello", said Laska.
# I turned and smiled.

echo '"Hello", said $name.\n I turned and stared"'
# "Hello", said $name.\n I turned and stared

Single-quotes create strings with strictly the characters you type between them; double-quotes allow for dynamic content and formatting.

You will commonly see that double-quotes are used “defensively” around every variable substitution like "$variable_name". Tools like ShellCheck will print warnings for variables that are used outside of quotes. This is to safeguard in the case that the string value of the variable has whitespace. After substitution, a variable value with whitespace will itself be interpreted as two or more strings. Wrapping it in double-quotes ensures it’s seen as a single string.

message="bon voyage"
count_characters $message
# bon: 3
# voyage: 6
count_characters "$message"
# bon voyage: 10

This is generally good advice and can prevent scripts breaking with whitespace in variables. But shouldn’t be used dogmatically. Too often I see that people use double-quotes every time a variable is referenced. For example, let’s say we want to print out the contents of a list of files with cat. We can use a for-loops, which loops once for each in a sequence of separate strings.

file_list="a.txt b.md c.tar"

# incorrect
for file in "$file_list"; do 
	cat $file
	# loops only once with the error:
	# cat: a.txt b.md c.tar: No such file or directory
done

# correct
for file in $file_list; do 
	cat $file
	# loops 3 times, printing the contents of each file
done

In the first loop, the variable is substituted within double-quotes, which is interpreted as a single string value with whitespace characters. In the second, the variable is substituted and interpreted as three separate strings.

Wrap up

Command-line shells are an indispensable tool in the life of anyone working with software, but I have yet to hear of anyone being formally taught how to understand shells. Discussing strings and variables may seem trivial, but in my experience, most bugs and errors in shell scripts and usage come from misunderstanding these fundamental pieces.

In my future post in this series, I will discuss the some of the most important strings in shells: environment variables and the PATH variable.