Say I have a large file with many rows and many columns. I'd like to find out how many rows and columns I have using bash.
-
give an example of the input and expected output– Diego Torres MilanoApr 23, 2011 at 0:12
-
Sorry, I'm not very familiar with bash. In R, it would look something like dim(input), which would return two numbers, #rows and #cols.– NickApr 23, 2011 at 0:14
-
an actual input file might look like: "blah\tdata\tdata\tdata\tdata\nblah2\tdata\tdata\tdata\tdata\n"– NickApr 23, 2011 at 0:15
-
I was hoping there might be an elegant way to do this with some built-in function...perhaps something like wc?– NickApr 23, 2011 at 0:16
13 Answers
Columns: awk '{print NF}' file | sort -nu | tail -n 1
Use head -n 1
for lowest column count, tail -n 1
for highest column count.
Rows: cat file | wc -l
or wc -l < file
for the UUOC crowd.
-
@Tim: Typo, should be
<
obviously. This isn't how I'd do it, but it satisfies the UUOC crowd (cat
enhances readability IMO, and I prefer it over the less readable pipes especially when answering newbie questions)– ErikApr 23, 2011 at 0:35 -
1@Erik You can also do "< file wc -l" and put the redirection before the command for enhanced readability. (Although, in this case I'm not sure why you don't just do "wc -l file") Apr 23, 2011 at 3:33
-
8No need for the sort or the tail, just do it all in awk: awk '{if( NF > max ) max = NF} END {print max}' Apr 23, 2011 at 3:36
-
1How to specify a delimiter here, for example if the file is tab separated– JoyApr 10, 2018 at 13:46
-
2@Joy:
awk -F'\t' '{print NF}' file | sort -nu | tail -n 1
to use tab as delimiter Dec 3, 2019 at 13:33
Alternatively to count columns, count the separators between columns. I find this to be a good balance of brevity and ease to remember. Of course, this won't work if your data include the column separator.
head -n1 myfile.txt | grep -o " " | wc -l
Uses head -n1
to grab the first line of the file.
Uses grep -o
to to count all the spaces, and output each space found on a new line. Uses wc -l
to count the number of lines.
EDIT: As Gaurav Tuli points out below, I forgot to mention you have to mentally add 1 to the result, or otherwise script this math.
-
2column count in a CSV will be
head -n1 myfile.txt | grep -o "," | wc -l
+ 1 because grep is counting the number of commas (or any other column separator) but the number of columns would be 1 more than that. Jun 14, 2021 at 15:43
If your file is big but you are certain that the number of columns remains the same for each row (and you have no heading) use:
head -n 1 FILE | awk '{print NF}'
to find the number of columns, where FILE is your file name.
To find the number of lines 'wc -l FILE' will work.
-
4
Little twist to kirill_igum's answer, and you can easily count the number of columns of any certain row you want, which was why I've come to this question, even though the question is asking for the whole file. (Though if your file has same columns in each line this also still works of course):
head -2 file |tail -1 |tr '\t' '\n' |wc -l
Gives the number of columns of row 2. Replace 2 with 55 for example to get it for row 55.
-bash-4.2$ cat file
1 2 3
1 2 3 4
1 2
1 2 3 4 5
-bash-4.2$ head -1 file |tail -1 |tr '\t' '\n' |wc -l
3
-bash-4.2$ head -4 file |tail -1 |tr '\t' '\n' |wc -l
5
Code above works if your file is separated by tabs, as we define it to "tr". If your file has another separator, say commas, you can still count your "columns" using the same trick by simply changing the separator character "t" to ",":
-bash-4.2$ cat csvfile
1,2,3,4
1,2
1,2,3,4,5
-bash-4.2$ head -2 csvfile |tail -1 |tr '\,' '\n' |wc -l
2
For rows you can simply use wc -l file
-l
stands for total line
for columns uou can simply use head -1 file | tr ";" "\n" | wc -l
Explanation
head -1 file
Grabbing the first line of your file, which should be the headers,
and sending to it to the next cmd through the pipe
| tr ";" "\n"
tr
stands for translate.
It will translate all ;
characters into a newline character.
In this example ;
is your delimiter.
Then it sends data to next command.
wc -l
Counts the total number of lines.
You can use bash. Note for very large files in terms of GB, use awk/wc
. However it should still be manageable in performance for files with a few MB.
declare -i count=0
while read
do
((count++))
done < file
echo "line count: $count"
If counting number of columns in the first is enough, try the following:
awk -F'\t' '{print NF; exit}' myBigFile.tsv
where \t
is column delimiter.
awk 'BEGIN{FS=","}END{print "COLUMN NO: "NF " ROWS NO: "NR}' file
You can use any delimiter as field separator and can find numbers of ROWS and columns
-
how to is it possible to put this as an
alias
? I tried different ways but it gives errors. Possibly related to the' "
.– ApexJun 10 at 15:14
Simple row count is $(wc -l "$file")
. Use $(wc -lL "$file")
to show both the number of lines and the number of characters in the longest line.
-
True. Silly of me to assume this was obvious:
wc -l file |cut -f 1
. Apr 23, 2011 at 0:26 -
@Tim Sylvester: You know UUOC is about the wasted process, right? I'm tempted to pass it back to you for that
cut
;)– ErikApr 23, 2011 at 0:28 -
How is it wasted if you want the file name removed from the output? Is there a way I don't know of to make
wc
not print the filename? Apr 23, 2011 at 0:31 -
Ah, using stdin doesn't show a filename. <headsmack> Now I feel dumb. Apr 23, 2011 at 0:35
head -1 file.tsv |head -1 train.tsv |tr '\t' '\n' |wc -l
take the first line, change tabs (or you can use ',' instead of '\t' for commas), count the number of lines.
Perl solution:
perl -ane '$maxc = $#F if $#F > $maxc; END{$maxc++; print "max columns: $maxc\nrows: $.\n"}' file
If your input file is comma-separated:
perl -F, -ane '$maxc = $#F if $#F > $maxc; END{$maxc++; print "max columns: $maxc\nrows: $.\n"}' file
output:
max columns: 5
rows: 2
-a
autosplits input line to @F
array
$#F
is the number of columns -1
-F,
field separator of , instead of whitespace
$.
is the line number (number of rows)
A very simple way to count the columns of the first line in pure bash (no awk, perl, or other languages):
read -r line < $input_file
ncols=`echo $line | wc -w`
This will work if your data are formatted appropriately.
Following code will do the job and will allow you to specify field delimiter. This is especially useful for files containing more than 20k lines.
awk 'BEGIN {
FS="|";
min=10000;
}
{
if( NF > max ) max = NF;
if( NF < min ) min = NF;
}
END {
print "Max=" max;
print "Min=" min;
} ' myPipeDelimitedFile.dat