Bash:Bash-Redirection

From Juneday education
Jump to: navigation, search


Meta information about this chapter

Expand using link to the right to see the full content.

Books this chapter belongs to

Introduction

This chapter introduces redirecting input and output.

Purpose

The purpose of this chapter is to give the students knowledge about using redirection using < and > so that they become more productive and more versatile when using input and output. This is essential when working in a UNIX-like environment where it is commonplace to save output from some commands in a log file, or using some file as the source of input to some command. Examples include setting up a database using SQL files and redirecting standard input from the files to the database management system, and, running test cases from files of input redirected to some test suite. Using redirects also works well as an introduction to pipes, in that it allows us to save output from one command in a temporary file and then using the file as input to the next command.

Goal

The goal of this chapter is that the students should be confident in using basic redirects to create and read files.

Instructions to the teacher

Common problems

It may be hard to understand that some programs (such as bc or sqlite) are interactive by default but switches to read input and process it non-interactively if there is a redirect (or pipe for that matter) in place changing the source of standard input.

An example:

$ cat arithmetic_exprs.txt
1+1
2+2
3+3
5*5
$ bc < arithmetic_exprs.txt
2
4
6
25
$ bc
bc 1.06.95
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
Now it's interactive!

While this might be counter-intuitive to students, it is important to understand this. It is a must for programming students who wish to be able to write flexible programs which operate in the same way (being able to use a redirect if present, and run interactively otherwise).

Chapter slides

Chapter videos

Introduction

Previously, we used a text editor to produce text files. In this exercise, we’ll see that we instead can redirect output from basically any tool or program to a file, as a way to create files on the fly. Text that is output from a program normally prints to a standard stream called stdout (or standard out). When we start the shell, that stream is connected to the terminal we are working with, so that output from a program is written in the terminal.

Exercises

Ex 01 - Setting up a directory for the chapter exercises

Let’s start by creating a directory for this session of exercises. Earlier you created the Ex00­text directory in your home directory. So why not make this directory called Ex00­streams?

$ cd ~
$ mkdir Ex00­streams
$ cd Ex00­streams

Ex 02 - Using touch to create empty files

We suggest watching

before doing this exercise.

Inside this (Ex00streams) directory, we’ll create ten empty files using shell expansion and the command touch. The touch command creates a new empty file when given the argument of a file that yet doesn’t exist. If you touch an existing file, the access time stamp of the file changes (like it has been touched by someone). The shell expansion using {} we actually saw in a previous exercise.

Running the following command will show you how it works:

$ touch file{0..9}

The idea was to create the files file0, file1, ...file9 using the {0..9} expression together with the command touch. Did it work? Let’s find out:

$ ls
file0 file1 file2 file3 file4 file5 file6 file7 file8 file9

Seems to have worked the way we hoped. Now, if we were to list all files whith name that starts with file, we could use the * as wildcard (this is called globbing, if anyone want’s to learn a new fancy name, or read more about it) where file* means "file-followed-by-anything":

$ ls file*

Did it work as expected, i.e. did it list all files whose name starts with "file"?

Ex 03 - Saving the result from ls to a file

We suggest watching: Redirects: (eng) before doing this exercise.

We can also tell ls explicitly to write every file that matches on a separate line. This is done with ls -­1 (minus one). The option ­1 means exactly that “one file per line”:

$ ls ­-1 file*

If we want to create a text file with the result from the command above, and we don’t want to write it manually using an editor, we can (finally!) tell the shell that standard out from ls should no longer be connected to the terminal. We do this using > (greater than sign) and write the name of the file to be created after it:

$ ls ­-1 file* > the_file_files

You can read that line like this: “run ls with the option ­-1 (dash one) and redirect all output to the file the_file_files”.

If there is no file called "the_file_files" in your current directory, it will be created. Note that if there is such a file, it will instead be overwritten.

Remember how to type the contents of a text file to the terminal? The answer is to use cat. So let’s investigate the contents of our the_file_files file (sorry about the dorky name):

$ cat the_file_files
file0
file1
file2
file3
file4
file5
file6
file7
file8
file9

This shows that the command $ ls ­-1 file* > the_file_files actually created a new file called the_file_files with the contents of the output from ls.

What we did there was to run ls but we didn’t want to see the result directly on the screen, so we told bash (the shell) to redirect all such output to the file with the beautiful name of the_file_files. Then we changed out mind and wanted so see what ended up in the the_file_files file (I’m starting to think the file name was poorly chosen), simply by using cat to print the file contents to the terminal.

The central part of the redirection to file was the > character. When used as above, it takes any (normal) output from a command and sends it to the file which is named after the > as in ls > result_of_ls.txt .

Ex 04 - saving the output from grep to a file

The redirection character > can be used with any command, so why not try it with grep?

Let's try to grep for something without redirection first. Let's say we wanted to list (for some reason) only files with an odd number (one of 1,3,5,7,9). We could use grep on the file the_file_files:

$ grep [13579] the_file_files
file1
file3
file5
file7
file9

The first argument to grep was a list of characters inside square brackets, and the second argument was the file to look in. It seemed to have worked. Using square brackets with grep could be though of as meaning "any of the characters in this list". Let’s run this command again but instead of printing the findings from grep to the screen, we’ll save it in a file called odds:

$ grep [13579] the_file_files > odds
$ cat odds
file1
file3
file5
file7
file9

Can you come up with a command line using grep to find only the lines with an even number?

See also: Grep

Ex 05 - commands reading input from a file

We suggest watching Redirecting stdin - read from file instead of keyboard: (eng) before doing this exercise.

Just like there is a standard stream for output (the one we used above), there is a standard stream for input. The standard input stream is what is used in interactive commands, i.e. commands where the program waits until you write something before anything interesting is happening.

If you remember the calculator command bc, you might also remember that we could use it interactively. Start bc and give it some arithmetic expression to calculate:

$ bc
bc 1.06.95
Copyright 1991­1994, 1997, 1998, 2000, 2004, 2006 Free Software
Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
9*9 (press enter after typing this expression)
81
quit
$

In the above example, you started bc without any options or arguments. It started in interactive mode and was waiting for you to give it something to do. You typed 9*9 (followed by Enter) and it replied with 81. You typed quit (followed by Enter) and you were back in the bash shell. This was an example of a program that read your input from the “standard in” stream. By default, stdin is connected to the “keyboard” you use to talk to your shell.

But the point in this section was to show that just as with the standard out stream, you may redirect also the standard in stream. This is done using the opposite redirection character, which is < and it means “use this file instead of the keyboard as standard input”.

Let’s create a file with a math expression, so that we can use that file as input to bc:

$ echo "9*9" > multi.txt

Remember, that command line meant: echo 9*9 but redirect the result not to the screen but instead to the file multi.txt. OK, so now we have a file with a mathematical expression. Let’s trick bc not to enter interactive mode, but to use that file as input rather than the “keyboard”:

$ bc < multi.txt
81

What did we just do? We told the shell that we wanted to run bc but please, use whatever is in the file multi.txt as standard input. The bc command accepted that and wrote 81 to the standard output (which still was connected to our terminal). The ambitious student would of course ask: “Can we both tell bc to read from a file and write the result to another file?”. The answer is, “Great question! Yes we can!”.

$ bc < multi.txt > product.txt
$ cat product.txt
81

We told bash that it should run bc with standard input redirected from the file multi.txt, and redirect any standard output from bc to the file product.txt. To see if it worked, we told cat to type the file to the standard output, and it showed us that the text 81 was inside product.txt . Read this section again if you didn't quite follow!

Solutions and comments

Expand using link to the right to see the full content.

Solution to Ex01

No solution needed (but we can't stress enough the practice of keeping your files organized in directories).

Solution to Ex 02

There isn't much to say in the solution sections here, but about touch we want to say that it is surprisingly useful! It is quite often handy to have a quick way to create empty files. Sometimes, it is the existence of files that are useful, and not the contents (as seen in the exercises were the files were there only to give ls something to work with). One case for creating empty files, is when testing software that checks the existence of files (but doesn't care about the contents).

Solution to Ex 03

Outputting the normal output from a command (the normal output is expected output excluding errors and diagnostics, also known as "the standard output stream or simply "standard out") to a file is as simple as that! Just add a > and a file name after the command.

Solution to Ex 04

If you wanted to grep for only even numbers, the argument to grep would be [0,2,4,6,8]. The command line (and resulting output) would read:

$ grep [0,2,4,6,8] the_file_files
file0
file2
file4
file6
file8

Solution to Ex 05

To let a command get all input from a file is done by adding < and the file name after the command. The input stream is called "standard input" or simply "standard in".

One application of redirecting standard in, is when we want to get results directly from an otherwise interactive program by supplying all input in a file, rather than typing it in by hand. We showed you how we could use bc in this way. Normally, bc (without redirection) would start interactively and wait for our input (from the keyboard) before it would do any calculations. Using redirection of standard in, it didn't have to wait and did the calculations directly and wrote the output to standard out.

Then we showed something that looked rather odd (perhaps): bc < multi.txt > product.txt. This was to show that you can combine the redirecting of standard in with the redirecting of standard out. The reason for showing this, was to show you that there are lots of ways to combine commands and files in bash. In a later chapter, we will show you one additional way to combine instead two commands using something called a pipe.

Links

External links

Where to go next

« PreviousBook TOCNext » (this is the last chapter)

See also

The next book to read should be Bash_Programming!