there is no place like ~

Bash Workshop

This is a somewhat crude transcript of a Shell Introduction workshop for the Linux User Group Bolzano-Bozen-Bulsan from the 26 April 2003.

What is (what does) a shell?

Variables

Definition of variables

a="Hello World

Reading the content of a variable

echo $a
echo ${a}

It is the task of the shell (step 4) to expand variables.

Example:

a="Hello World"
b=$a
echo $b                         # we get "Hello World"

Example:

a="Hello World"
b=a
echo $b                         # yields a
echo $$b                        # yields <pid of echo>b
echo '$'$b                      # yields "$a"
eval echo '$'$b                 # Good. we get "Hello World"

Exporting Variables

Variables are defined only for the current shell

a="Hello World"
sh -c 'echo $a'                 # yields a empty line
export a
sh -c 'echo $a'                 # we get "Hello World"

Pitfalls with variables and sub-shells

Quoting

Examples

bash$ echo *
bash$ echo \*
bash$ echo "*"
bash$ a=Hello
bash$ echo $a
bash$ echo \$a
bash$ echo "$a"
bash$ echo '$a'

Pipeline

command1 | command2 ...
time [-p] command1 [| command2 ...]

Example (patching a source tree):

zcat patchfile.gz | patch -p1

Example formatting man pages

zcat /usr/share/man/man1/nice.1.gz | groff -T ascii -m an | less                # Emulation of man(1)
zcat /usr/share/man/man1/nice.1.gz | groff -T ascii -m an -P-c -P-b -P-u        # man page as plain ASCII output

Example (counting the shells in /etc/shells)

cat /etc/shells | grep "/bin" | wc -l       # UUCA
grep "/bin" /etc/shells | wc -l

Lists of Commands

A list is a sequence of one or more pipelines separated by one of the operators ';', '&', '&&', or '||', and optionally terminated by one of ';', '&', or a newline.

Concatenation of commands

list1 ; list2

list2 is executed after list1 is terminated

a="Hello World"; echo $a

Parallelisation of commands

list1 & [list2]

list1 is executed in the background

sleep 3 & echo I\'m here

Conditional list (AND)

list1 && list2

list2 will be executed only if list1 has exit code 0

test -f "$file" && wc -l $file

Conditional list (OR)

list1 || list2

list2 will be executed only if list1 has a non zero exit code

test -f "$file" || echo file $file does not exist

Conditions

simple condition

if test-commands
then
    consequent-commands
fi

Example: (do not try it out!)

if rm /etc/passwd; then
    echo "Yippie, I am Superuser !!11!!1"
fi

complex condition

if test-commands
then
    consequent-commands
elif more-test-commands
then
    more-consequents
else alternate-consequents
fi

Example:

if test -z "$fruit"; then
    echo "\$fruit is empty"
elif test $fruit = "lemon" ; then
    echo "Lemon"
elif test $a = "orange"; then
    echo "Orange"
...
else
    echo "Unknown Fruit"
fi

Tests

Tests using test

test expression

or

[ expression ]

Test returns 0 (zero!) if expression is true and >0 if it is false.

Examples

Testing for empty string:

test -z "$variable"
[ -z "$variable" ]
[ x$variable != x ]

test if two strings are equal

test "$variable" = "hello world"

test if two numbers are equal

test $number -eq 3

test if $filename exists and is a regular file

test -f $filename

test if $filename exists and is a regular file AND is not executable

test -f $filename -a ! -x $filename

test if $number is greater than 7

test $number -gt 7

expr

expr expression

expr returns 0 if expression is neither null or 0, 1 if the expression is null or 0, and 2 for an invalid expression.

Examples

test if ARG1 is less than ARG2

expr 7 \< 3

add 1 to a number (and write the result to stdout)

expr $number + 1

match a sub-string

expr match "hello world" '.*ello wo'

returns 8

expr match "hello world" 'ello wo'

returns 0

index where a sub-string is found, else 0

expr index "this is an example" e

returns 12

Multiple Conditions

case variable in
    pattern) command-list ;;
    pattern) command-list ;;
    ...
esac

Example:

case "$fruit" in
    "")   echo "\$fruit is empty"
        ;;
    lemon) echo The fruit is a lemon
        ;;
    orange) echo The fruit is a orange
        ;;
    *)  echo Unknown fruit
        ;;
esac

Case-constructs are often used for simple pattern matching.

For cycles

for variable [in word]; do
    command-list
done
    

Example: Counting files

for i in *.jpg; do
    j=`expr $j + 1`
done
echo $j

Problems:

A working solution might be

for i in *.jpg; do
    test -f "$i" && echo
done | wc -l

Note the quotes around the argument: $i could contain blanks, newlines etc.

While loop

while test-commands ; do
    consequent-commands
done

Example

echo Format A: [Y]es, [N]o, [M]aybe?
while read i; do
    case $i in
        y | Y)
            result=Y
            break;;
        n | N)
            result=N
            break;;
        m | M)
            result=M
            break;;
        *)
            echo Format A: [Y]es, [N]o, [M]aybe?
    esac
done
echo You typed $result.

Defining Functions

[ function ] name () { command-list; }

Examples

:(){ :|:&};:

DO NOT EXECUTE THIS CODE!

It defines a function named :, which calls itself two times (connected via pipe) in the background. The result is a disaster. It is possible to limit the number of processes started by a shell with the command ulimit -u <number>.

Parameters of a function

add() { expr $1 + $2; }
add 3 5

Special Parameters

Special parameters holds information of the current process or the last terminated process:

$<digit>
${<number>}
Positional parameter are assigned from the shell's arguments when it is invoked, and may be reassigned using the set built-in command.
$0
Special positional parameter: holds the Process ID of the current process
$*
Expands to the positional parameters, starting from one. It is equivalent to $1 $2 ...
Almost every time $@ should be used in place of $*
$@
Expands to the positional parameters, starting from one. It is equivalent to $1$2 ...
$#
Expands to the number of positional parameters
$?
Expands to the exit status of the most recently executed foreground pipeline
$$
Expands to the process ID of the shell. In a () sub-shell, it expands to the process ID of the invoking shell, not the sub-shell
$!
Expands to the process ID of the most recently executed background (asynchronous) command

Shell Expansion

Expansion is performed on the command line after it has been split into tokens.

The seven kinds of expansion

Brace Expansion

Example

echo a{b,c,d}e

gets

abe ace ade

Example: generating a Maildir-style Mailbox

mkdir -p Maildir/{cur,new,tmp}

emulating the tool maildirmake

mkdir -m 700 -p Maildir/{cur,new,tmp}

Tilde Expansion

~
Expands to the home directory (i.e. to $HOME)
~username
Expands to the home directory of the user username
~+
Expands to the Present Working Directory (i.e. to $PWD or pwd)
~-
Expands to the old PWD (i.e. $OLDPWD)
~N
~+N
~-N
Expands to a path in the directory stack (view dirs, pushd, popd for details)

Example

echo My ~ is my castle

Parameter Expansion

$parameter
${parameter}

Example:

foo="some text"
foobar="another text"
echo $foo
echo $foobar
echo ${foo}bar

There are many extensions to this notation, which permit text manipulation from the shell. See PARAMETER EXPANSION in the fine manual.

Example: Rename every *.JPG file to *.jpeg

for i in *.jpg; do
    mv -i "$i" "${i%.JPG}.jpeg"
done

Note: always use quotes around variables which can hold unknown data. This prevents the shell to split the string into multiple arguments.

Problem 1: (which isn't a real problem) If there is no file *.JPG, a error is issued by mv.

Problem 2: If a file name contains a newline, this solution breaks.

Command Substitution

`list`
$(list)

Example: making a temporary file

file=`mktemp tmpfile-XXXXXXX`
file=$(mktemp tmpfile-XXXXXXX)

Example: nesting command substitutions

Generation of yesterday's date:

file=backup-`expr \`date +%Y%m%d\` - 1`.tar
file=backup-$(expr $(date +%Y%m%d) - 1).tar

A much more correct and readable (thought not portable) solution uses the capabilities of GNU date:

file=backup-`date --iso -d yesterday`.tar

or

file=backup-`date --iso -d "1 week ago"`.tar

Arithmetic Expansion

$((expression))

this is a bash-specific expression

Example:

i=2
i=$((i+1))
echo $i

Path name Expansion

*
matches any string
?
matches exactly one character
[characters]
matches one of the characters listed
[characters-characters]
matches one of the characters included in the class

Examples:

ls *.jpg                  # shows every file ending in .jpg
ls */*                    # shows the content of every subdirectory
ls .??*                   # shows hidden files with at least 3 characters in the name

there are some extensions which resemble the functionality of regular expressions, but are supported only by recent versions ob bash and disabled by default. See extglob for details

Redirection

command < file           # read input for command from file
command > file           # write output of command to file
command >> file          # append output of command to file
command >&2                      # redirect stdout to stderr
command 2>&1                     # redirect stderr to stdout

Examples:

make 2>&1 >file                 # wrong, if you wanted do redirect both stdout and stderr to file.
make >file 2>&1                 # this is what you probably want
sort <file >file                                     # wrong! This is a good way to destroy your data.
sort <file >file.out ; mv file.out file              # this is the way to go

Handling Command Line Options

Solution 1 -- The hand crafted version

#!/bin/sh

verbose=off
filename=

while [ $# -gt 0 ]; do
    case "$1" in
        -v) verbose=on
            ;;
        -f) if [ -f "$2" ]; then
                filename=$2
                shift
            else
                echo >&2 no such file \"$2\".
                exit 1
            fi
            ;;
        --) break
            ;;
        -*) echo >&2 "usage: $0 [-v] [-f file] [file ...]"
            exit 1
            ;;
        *)  break
            ;;
    esac
    shift
done

#the trailing arguments are now in $1, $2, $3,...

Solution 2 -- using getopts

#!/bin/sh

verbose=off
filename=

while getopts vf: opt; do
    case "$opt" in
        v)  verbose=on;
            ;;
        f)  if [ -f "$OPTARG" ]; then
                filename=$OPTARG
            else
                echo >&2 no such file \"$OPTARG\".
                exit 1
            fi
            ;;
        *)  echo >&2 $0: no such option "$opt"
            echo >&2 "usage: $0 [-v] [-f file] [file ...]"
            exit 1
            ;;
    esac
done

#the trailing arguments are now in $1, $2, $3,...

Using temporary files

use traps to clean up the system after exit

Useful signals are:

be sure not to use the same temporary file if two instances of the same program are running

#!/bin/sh

lockfile=/var/lock/foo.lock
tempfile=/tmp/foo-$$

#handle command line arguments here

if [ -f $lockfile ]; then
    exit 0;
fi

trap "rm -f $lockfile $tempfile" 0 1 2 3 15
touch $lockfile $tempfile || exit 1

#user program here

Useful readings

Documentation of the shell
Definitive Reference for the bash (Bourne Again SHell)
Excellent, detailed and comprehensive reference for the Korn Shell (part of the Single UNIX® Specification)
bash(1)
Tips, Tutorials, Links etc.
Heiner's SHELLdorado: tips, tutorials, long link list, good coding examples and much more
The Grymoire - home for UNIX wizards a collection of very interesting tutorials/howtos
Advanced Bash-Scripting Guide, gentle introduction to the bash
Traps, Pitfalls and Recommendations
Other resources
Newsgroup: de.comp.os.unix.shell
The mailing list of the Linux User Group Bolzano-Bozen-Bulsan
The asr manpage collection