953

I have the following .txt file:

Marco
Paolo
Antonio

I want to read it line-by-line, and for each line I want to assign a .txt line value to a variable. Supposing my variable is $name, the flow is:

  • Read first line from file
  • Assign $name = "Marco"
  • Do some tasks with $name
  • Read second line from file
  • Assign $name = "Paolo"
1
  • 4
    Can those questions maybe be merged somehow? Both have some really good answers that highlight different aspects of the problem, the bad answers have in-depth explanations in the comments what's bad about them, and as of now you cannot really get a whole overview on what to consider, from the answers of one single question from the pair. It would be helpful to have all of it in one spot, rather than splotted over 2 pages.
    – Egor Hans
    Nov 12, 2017 at 16:11

10 Answers 10

1706

The following reads a file passed as an argument line by line:

while IFS= read -r line; do
    echo "Text read from file: $line"
done < my_filename.txt

This is the standard form for reading lines from a file in a loop. Explanation:

  • IFS= (or IFS='') prevents leading/trailing whitespace from being trimmed.
  • -r prevents backslash escapes from being interpreted.

Or you can put it in a bash file helper script, example contents:

#!/bin/bash
while IFS= read -r line; do
    echo "Text read from file: $line"
done < "$1"

If the above is saved to a script with filename readfile, it can be run as follows:

chmod +x readfile
./readfile filename.txt

If the file isn’t a standard POSIX text file (= not terminated by a newline character), the loop can be modified to handle trailing partial lines:

while IFS= read -r line || [[ -n "$line" ]]; do
    echo "Text read from file: $line"
done < "$1"

Here, || [[ -n $line ]] prevents the last line from being ignored if it doesn't end with a \n (since read returns a non-zero exit code when it encounters EOF).

If the commands inside the loop also read from standard input, the file descriptor used by read can be chanced to something else (avoid the standard file descriptors), e.g.:

while IFS= read -r -u3 line; do
    echo "Text read from file: $line"
done 3< "$1"

(Non-Bash shells might not know read -u3; use read <&3 instead.)

16
  • 34
    There is a caveat with this method. If anything inside the while loop is interactive (e.g. reads from stdin), then it will take its input from $1. You will not be given a chance to enter data manually.
    – carpie
    Jan 16, 2014 at 16:25
  • 17
    Of note - some commands break (as in, they break the loop) this. For example, ssh without the -n flag will effectively cause you to escape the loop. There's probably a good reason for this, but it took my a while to nail down what was causing my code to fail before I discovered this.
    – Alex
    May 20, 2014 at 21:22
  • 8
    as a one-liner: while IFS='' read -r line || [[ -n "$line" ]]; do echo "$line"; done < filename
    – Jo Jo
    Dec 2, 2015 at 23:20
  • 12
    @OndraŽižka, that's caused by ffmpeg consuming stdin. Add </dev/null to your ffmpeg line and it won't be able to, or use an alternate FD for the loop. That "alternate FD" approach looks like while IFS='' read -r line <&3 || [[ -n "$line" ]]; do ...; done 3<"$1". Dec 14, 2016 at 16:11
  • 12
    grumble re: advising a .sh extension. Executables on UNIX don't typically have extensions at all (you don't run ls.elf), and having a bash shebang (and bash-only tooling such as [[ ]]) and an extension implying POSIX sh compatibility is internally contradictory. May 21, 2017 at 21:32
347

I encourage you to use the -r flag for read which stands for:

-r  Do not treat a backslash character in any special way. Consider each
    backslash to be part of the input line.

I am citing from man 1 read.

Another thing is to take a filename as an argument.

Here is updated code:

#!/usr/bin/bash
filename="$1"
while read -r line; do
    name="$line"
    echo "Name read from file - $name"
done < "$filename"
7
  • 7
    Trims leading and trailing space from the line
    – barfuin
    Sep 21, 2014 at 13:25
  • @Thomas and what happens to the spaces in the middle? Hint: Unwanted attempted command execution.
    – kmarsh
    Feb 23, 2016 at 21:42
  • 1
    This worked for me, in contrast to the accepted answer. Jun 30, 2016 at 14:15
  • 3
    @TranslucentCloud, if this worked and the accepted answer didn't, I suspect that your shell was sh, not bash; the extended test command used in the || [[ -n "$line" ]] syntax in the accepted answer is a bashism. That said, that syntax actually has pertinent meaning: It causes the loop to continue for the last line in the input file even if it doesn't have a newline. If you wanted to do that in a POSIX-compliant way, you'd want || [ -n "$line" ], using [ rather than [[. Dec 14, 2016 at 17:25
  • 3
    That said, this does still need to be modified to set IFS= for the read to prevent trimming whitespace. Dec 14, 2016 at 17:28
161

Using the following Bash template should allow you to read one value at a time from a file and process it.

while read name; do
    # Do what you want to $name
done < filename
13
  • 20
    as a one-liner: while read name; do echo ${name}; done < filename
    – Jo Jo
    Dec 2, 2015 at 23:17
  • 6
    @CalculusKnight, it only "worked" because you didn't use sufficiently interesting data to test with. Try content with backslashes, or having a line that contains only *. Jun 28, 2016 at 23:42
  • 8
    @Matthias, assumptions that eventually turn out to be false are one of the largest sources of bugs, both security-impacting and otherwise. The largest data loss event I ever saw was due to a scenario someone assumed would "literally never come up" -- a buffer overflow dumping random memory into a buffer used to name files, causing a script that made assumptions about which names could possibly ever occur to have very, very unfortunate behavior. Dec 14, 2016 at 15:59
  • 5
    @Matthias, ...and that's especially true here, since code samples shown at StackOverflow are intended to be used as teaching tools, for folks to reuse the patterns in their own work! Dec 14, 2016 at 16:06
  • 6
    @Matthias, I utterly disagree with the claim that "you should only devise your code for data you expect". Unexpected cases are where your bugs are, where your security vulnerabilities are -- handling them is the difference between slapdash code and robust code. Granted, that handling doesn't need to be fancy -- it can just be "exit with an error" -- but if you have no handling at all, then your behavior in unexpected cases is undefined. Jun 8, 2017 at 20:26
97
#! /bin/bash
cat filename | while read LINE; do
    echo $LINE
done
11
  • 8
    Nothing against the other answers, maybe they are more sofisticated, but I upvote this answer because it's simple, readable and is enough for what I need. Note that, for it to work, the text file to be read must end with a blank line (i.e. one needs to press Enter after the last line), otherwise the last line will be ignored. At least that is what happened to me. Feb 16, 2016 at 13:32
  • 14
    Useless use of cat, shurely ? Apr 8, 2016 at 11:42
  • 5
    And the quoting is broken; and you should not use uppercase variable names because those are reserved for system use.
    – tripleee
    Jun 7, 2016 at 11:35
  • 8
    @AntonioViniciusMenezesMedei, ...moreover, I've seen folks sustain financial losses because they assumed these caveats would never matter to them; failed to learn good practices; and then followed the habits they were used to when writing scripts that managed backups of critical billing data. Learning to do things right is important. Jun 28, 2016 at 23:46
  • 7
    Another problem here is that the pipe opens a new subshell, i.e. all variables set inside the loop can't be read after the loop finished.
    – mxmlnkn
    Sep 10, 2017 at 19:18
26

Use:

filename=$1
IFS=$'\n'
for next in `cat $filename`; do
    echo "$next read from $filename" 
done
exit 0

If you have set IFS differently you will get odd results.

10
  • 35
    This is a horrible method. Please don't use it unless you want to have problems with globbing that will take place before you realize it! May 20, 2014 at 12:28
  • 1
    This is not horrible, no break in the execution. Mar 17, 2015 at 14:56
  • 16
    @MUYBelgium did you try with a file that contains a single * on a line? Anyway, this is an antipattern. Don't read lines with for. Oct 14, 2015 at 16:44
  • 3
    @OndraŽižka, the read approach is the best-practices approach by community consensus. The caveat you mention in your comment is one that applies when your loop runs commands (such as ffmpeg) that read from stdin, trivially solved by using a non-stdin FD for the loop or redirecting such commands' input. By contrast, working around the globbing bug in your for-loop approach means making (and then needing to reverse) shell-global settings changes. Dec 14, 2016 at 16:14
  • 2
    @OndraŽižka, ...moreover, the for loop approach you use here means that all content has to be read in before the loop can start executing at all, making it entirely unusable if you're looping over gigabytes of data even if you have disabled globbing; the while read loop needs to store no more than a single line's data at a time, meaning it can start executing while the subprocess generating content is still running (thus being usable for streaming purposes), and also has bounded memory consumption. Dec 14, 2016 at 16:18
22

Many people have posted a solution that's over-optimized. I don't think it is incorrect, but I humbly think that a less optimized solution will be desirable to permit everyone to easily understand how is this working. Here is my proposal:

#!/bin/bash
#
# This program reads lines from a file.
#

end_of_file=0
while [[ $end_of_file == 0 ]]; do
  read -r line
  # the last exit status is the 
  # flag of the end of file
  end_of_file=$?
  echo $line
done < "$1"
12

If you need to process both the input file and user input (or anything else from stdin), then use the following solution:

#!/bin/bash
exec 3<"$1"
while IFS='' read -r -u 3 line || [[ -n "$line" ]]; do
    read -p "> $line (Press Enter to continue)"
done

Based on the accepted answer and on the bash-hackers redirection tutorial.

Here, we open the file descriptor 3 for the file passed as the script argument and tell read to use this descriptor as input (-u 3). Thus, we leave the default input descriptor (0) attached to a terminal or another input source, able to read user input.

1
  • if you want to accepted piped input, exec 3<&0 Dec 29, 2021 at 20:43
7

For proper error handling:

#!/bin/bash

set -Ee    
trap "echo error" EXIT    
test -e ${FILENAME} || exit
while read -r line
do
    echo ${line}
done < ${FILENAME}
3
  • Could you please add some explanation? Apr 10, 2018 at 18:28
  • Unfortunately it misses the last line in the file.
    – ungalcrys
    Apr 19, 2018 at 7:15
  • ...and also, on account of the lack of quoting, munges lines that contain wildcards -- as described in BashPitfalls #14. Nov 29, 2018 at 22:22
1

Use IFS (internal field separator) tool in bash, defines the character using to separate lines into tokens, by default includes <tab> /<space> /<newLine>

step 1: Load the file data and insert into list:

# declaring array list and index iterator
declare -a array=()
i=0

# reading file in row mode, insert each line into array
while IFS= read -r line; do
    array[i]=$line
    let "i++"
    # reading from file path
done < "<yourFullFilePath>"

step 2: now iterate and print the output:

for line in "${array[@]}"
  do
    echo "$line"
  done

echo specific index in array: Accessing to a variable in array:

echo "${array[0]}"
2
  • you need to quote your variables, array[i]="$line"
    – ErikE
    Jun 10, 2020 at 22:28
  • array[i++]=$line without the need for let
    – Jetchisel
    Jul 7, 2020 at 23:37
-9

The following will just print out the content of the file:

cat $Path/FileName.txt

while read line;
do
echo $line     
done
1
  • 6
    This answer really doesn’t add anything over existing answers, doesn’t work due to a typo/bug, and breaks in many ways. May 21, 2019 at 8:59

Not the answer you're looking for? Browse other questions tagged or ask your own question.