Bash shell basics — pipe and redirection

Compose pipes and pipelines

Bash shell pipes are awesome. Pipe the output of one command into the input of another. The piped commands create a pipeline.

A ‘pipeline’ is a sequence of one or more commands separated by one of
the control operators ‘|’ or ‘|&’.

A pipeline could be a shell application in it’s own right, and you can pipe a number of pipelines together to create another shell application.

Then you can conceptually take the same elements and combine them a different way to create another shell application.

A simple shell application could start with a list of something, then pipe into a filter or two, then pipe into a sort to finish up.

# List directories (use {} to select multiple items)
# then filter by column, then sort.
ls -ld vimfiles/{colors,plugin} | awk '{print $9}' | sort

Another example:

# List files, then filter by row, then filter by column, then sort.
ls -l | grep drw | awk '{print $9}' | sort

So the Bash shell makes it easy to pipe the output of one command into the input of another.

But the concept of feeding the output of a command (or pipeline) into the input of another doesn’t stop there….

Standard input and standard output

A command can operate without arguments. The most basic is the cat command.

# Copy from standard input (ie keyboard) to standard output (ie screen).
cat

The basic cat command copies standard input to standard output. This means that when you execute the command, what you type on the keyboard (standard input) is echoed to the screen (standard output).

So type in ‘hello’ from the keyboard and ‘hello’ is echoed back to the screen.

But if you hardcode an argument (ie a filename) to cat, then the command will not read the standard input. Instead the input is redirected to become the contents of a file, and so now the contents of the file specified is input and echoed on the screen (standard output).

# Now the cat command will readily accept an
# argument as an input (eg file.txt file) instead of a standard input.
cat file.txt

Note: Output will still be to standard output (ie screen).

But not all commands have this auto ability to input from the keyboard or redirect input from a file contents.

For commands that accept hard coded arguments, there are other ways to get an input.

More standard input and standard output

Let’s examine the ls command. It doesn’t natively take standard input from the keyboard.

# List files of a directory.
ls

But ls does take arguments that are hard coded.

# List the files of a directory, and also take an argument as an input.
ls -l

Note: Output will still be to standard output (ie screen).

So in the example above, the ls command expects the first argument to be input as an option. The option is specified as ‘-l’, and is effectively hard coded.

It’s easy to add more arguments as an input, and they are also hard coded.

# list the files of a directory, and also take 2 arguments for input
ls -l Downloads

Note: Output will still be to standard output (ie screen).

Any argument you specify for this command is typically hard coded as an input.

But what if you don’t want to hard code the argument, and instead make the argument as the redirected input of the output of a pipeline?

This is where input redirection comes in using the ‘<’ operator.

Input redirection

So examine a simple pipeline, where the output of ls is piped into the input of grep.

We can just list the contents of a directory, and filter to get the sub-directories only.

# Grep will take an argument as the first input, 
# then take the output of a pipe of ls as the second input.
ls -l | grep drw

Note: Output will still be to standard output (ie screen).

Now lets do a similar command, but this time with input redirection. The ‘<’ operator is going to be used to signify that the input is not hard coded, but instead is going to be the output of a pipeline/command.

# In fact, grep can take arguments, pipes, and redirection as inputs.
grep drw <(ls -l)

Note: Output will still be to standard output (ie screen).

Redirection can be used with any command, but let’s re-examine the cat command again.

This time we’ll using the cat command with a redirected input, plus a redirected output using the ‘>’ operator.

# In fact, as inputs, the cat command will also take a redirected pipe.
cat <(ls -l| grep drw) >test.txt

Also the output can be redirected away from standard output to a file.

So now the output is redirected to a file instead of to the screen (standard output). Then we can use cat to echo the contents of the file to the screen.

Another input redirection

This time let’s pipe the output of a pipeline into a text editor. eg gvim.

One way to do this is just to tell gvim to use the standard input as an input. ie with the dash ‘-’ operator.

# Another pipe, but this time as in input to gvim.
# Note: dash '-' tells gvim to expect standard input as the input
(ls -l | grep drw) | gvim -

But we can also use the ‘<’ operator to redirect the input into gvim from a pipeline.

# Another redirection, and this time redirect the output of a pipeline
# as an argument of gvim
gvim <(ls -l | grep drw)

When the gvim editor opens, then we see the output of the pipeline was redirected as the input.

There you are — use pipelines and redirection to make your next shell application the best yet!

Reference

https://workaround.org/linuxtip/pipes

Checkout some other Bash articles here:

Bash shell basics — lists of pipelines asynchronously

Bash shell basics — scoping

Bash shell basics — inline execution

Bash shell basics — synchronous or asynchronous