How to Read a File Line By Line in Bash

Introduction

When writing Bash scripts, you will occasionally come across circumstances where you must read a file line by line. You may, for example, have a text file with data that the script should process.

In this tutorial, we'll go over how to read a file line by line in Bash. We will also address a few FAQs on how to read a file line by line in Bash.

Reading a File Line By Line Syntax

The following is the most basic syntax for reading a file line by line:

while IFS= read -r line; do
  printf '%s\n' "$line"
done < input_file

Alternatively, you may use the single-line version:

while IFS= read -r line; do printf '%s\n' "$line"; done < input_file

How does it work?

The name of the file routed to the while loop is the input file (input_file). The read command reads each line of the file and assigns it to the line variable. The while loop ends when all lines have been processed.

The read command sees the backslash as an escape character and removes all leading and trailing white spaces by default, which can lead to unexpected results. The command is run with the -r option to deactivate backslash escaping, and the internal field separator (IFS) is cleared to disable trimming.

To make the code more portable and avoid undesirable behaviour, we're using [printf] instead of echo. If the line contains values like "-e," for example, it will be interpreted as an echo option.

Reading a File Line By Line Examples

Take a look at the following example. Let's say we have a file called distros.txt that has a list of some of the most popular Linux distributions, separated by commas (,):

Ubuntu,apt
Debian,apt
CentOS,yum
Arch Linux,pacman
Fedora,dnf

Run the following code in your terminal to read the file line by line:

while IFS= read -r line; do
  printf '%s\n' "$line"
done < distros.txt

The code reads each line of the file, assigns it to a variable, then prints it out. Basically, the output would be the same as if you used the cat program to display the file content.

What if you only want to print apt-based distributions? One method is to verify if the line includes the apt substring using the if statement:

while IFS= read -r line; do
  if [[ "$line" == *"apt"* ]]; then
    printf '%s\n' "$line"
  fi
done < distros.txt

Output

Ubuntu,apt
Debian,apt

You can also supply multiple variables to the read command while reading a file line by line, which will split the line into fields depending on IFS. The first field goes with the first variable, the second with the second, and so on. If the number of fields exceeds the number of variables, the remaining fields are assigned to the last variable.

In the example below, we set IFS to a comma (,) and supply two variables to the read command: distro and pm. The first variable (distro) will be allocated to everything from the beginning of the line until the first comma, and the rest of the line will be assigned to the second variable (pm):

while IFS=, read -r distro pm; do
  printf '%s is the package manager for %s\n' "$pm" "$distro"
done < distros.txt

Output

apt is the package manager for Ubuntu
apt is the package manager for Debian
yum is the package manager for CentOS
pacman is the package manager for Arch Linux
dnf is the package manager for Fedora

Alternative File Reading Methods

Using a Process Substitution

Process substitution is a feature that allows you to save command output as a file:

while IFS= read -r line; do
  printf '%s\n' "$line"
done < <(cat input_file )

Using a Here String

Here String is a Here document variation. The newlines are kept in the string (cat input file):

while IFS= read -r line; do
  printf '%s\n' "$line"
done <<< $(cat input_file )

Using File descriptor

You can also use a file descriptor to provide input to the loop:

while IFS= read -r -u9 line; do
  printf '%s\n' "$line"
done 9< input_file

To avoid conflicts with shell internal file descriptors, use a number between 4 and 9 when working with file descriptors.

FAQs to Read a File Line By Line in Bash

What does the `IFS= read -r line` command do?

In Bash, IFS stands for Internal Field Separator, and setting it to blank (IFS=) ensures leading/trailing whitespaces are preserved. The -r option of read is used to prevent backslashes from being interpreted.

How can I process each line read from a file in my script?

Within the while loop, you can perform any desired actions on each line, such as storing it in a variable, parsing data, or performing calculations.

How do I determine if I have reached the end of the file?

The read command returns a non-zero exit code when it reaches the end of the file. You can use this condition to break out of the loop.

How can I skip empty lines while reading a file?

You can check if a line is empty and skip it using an if statement within the while loop. Example: if [[ -z "$line" ]]; then continue; fi

Is it possible to process a specific range of lines in a file?

Yes, you can use a counter variable inside the loop and break when reaching the desired line number. This allows you to process a specific range of lines.

Can I read files from a directory recursively?

Yes, you can use the find command to search for files recursively, and then loop over each file to read it line by line.

What if my file contains special characters or spaces in the filename?

To handle filenames with spaces or special characters, enclose the filename in double quotes within the loop. Example: while IFS= read -r line; do echo "$line"; done < "file name.txt"