Bash shell basics — lists of pipelines asynchronously

The Bash shell is incredibly powerful. And as a shell, it’s capabilities are often under-appreciated.

Functions in the Bash shell are first class citizens, just like in most other languages.

But in a lot of ways, the fundamentals of the Bash shell are pipelines.

Pipelines

A pipeline could be just a single command, or a sequence of many commands.

A ‘pipeline’ is a sequence of one or more commands separated by one of
the control operators ‘|’ or ‘|&’.

A common pipeline pattern is to execute a command, then filter, then sort the output of that command.

An example of a pipeline is below:

# pipe the output of one command into the input of another
ps -eo pid,euser,start,etime,args | grep -e root| sort

The syntax of a pipeline can be adjusted too.

The output of the ps command can be piped into pgrep, which will retain the column headings in the output.

# Pipe an output into an input
ps -o pid,euser,start,etime,args --sort $(pgrep -u root)

It may also be better to do the sorting earlier, as part of the ps command, rather than at the end of the pipeline.

Lists of pipelines

Just as a pipeline is a sequence of one or more commands; a list is a sequence of one or more pipelines.

A ‘list’ is a sequence of one or more pipelines

A simple example of a list using the ‘;’ separator is below:

# execute pipeline sequentially regardless of error
mkdir test; cd test

Using the ‘;’ separator, the 2nd pipeline will execute regardless of the error status of the 1st pipeline.

Can also put in conditions to only run the 2nd pipeline, only if the 1st pipeline didn’t return an error. In this case, we use a ‘&&’ as the separator.

# if the 1st pipeline has no error, then execute the 2nd pipeline.
mkdir ~/test && cd ~/test

The first pipeline needs to be successful before the second pipeline can execute.

A list of pipelines can be executed in two ways.

• execute in a subshell
• execute in the current shell

Execution in a subshell is performed using the ( and ) characters and is as follows:

# execute a list of pipelines in a subshell.
(echo 'Hello wally'; echo 'Hello wally world')

# as above
(echo 'Hello wally'
echo 'Hello wally world')

Execution in the current shell is performed using the ‘{space’ and ‘;space}’ characters and is perhaps more conventional:

# execute list of pipelines in the current shell.
# Note: the use of space characters are important
{ echo 'Hello wally'; echo 'Hello wally world'; }

# as above
{ echo 'Hello wally'
echo 'Hello wally world'; }

Any errors in the pipeline, will be reported in the pipestatus environment variable.

# query for execution status (0=no error, !0=error) of the last pipeline
echo $PIPESTATUS

A status of zero means no error occurred in the last pipeline.

A non-zero value will be returned if there is an error.

Lists of pipelines synchronously

Can also process many lists of pipelines synchronously like the Bash script below:

#!/bin/bash

cm="date; hostname; uname -a"
# execute many lists of pipelines
for server in 192.168.0.1 192.168.0.2 192.168.0.3
do
    # synchronous execution of each list of pipelines
    ssh root@$server $cm; echo 'pipeline status: '$PIPESTATUS
done

exit

As each pipeline encounters time-outs, then the status is non-zero.

It is also possible to process lists of pipelines asynchronously too.

Lists of pipelines asynchronously

Much quicker than running many lists of pipelines synchronously, is to instead run the lists asynchronously.

Asynchronous execution of a list of pipelines will each will run as a separate job.

Asynchronous lists of pipelines are created using a pair of ‘(‘ and ‘)’ followed by ‘&’. The ‘&’ is used to create a new job for each list.

Then follow up with a ‘wait’ command, to wait for all the jobs to finish before exiting the script.

#!/bin/bash

cm="date; hostname; uname -a"
# execute many lists of pipelines
for server in 192.168.0.1 192.168.0.2 192.168.0.3
do
    # asynchronous execution of each list of pipelines
    (ssh root@$server $cm; echo 'pipeline status: '$PIPESTATUS)&
done

echo
echo 'wait for completion of the jobs...'
echo

wait

exit

The Bash script completes earlier, as the jobs are running in asynchronously.

Checkout some other Bash articles here:

Bash shell basics — scoping

Bash shell basics — inline execution

Bash shell basics — synchronous or asynchronous

Bash shell basics — pipe and redirection