Questions tagged [awk]

AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool. AWK is used largely with Unix systems.

Filter by
Sorted by
Tagged with
915 votes
37 answers
765k views

How can I do a recursive find/replace of a string with awk or sed?

How do I find and replace every occurrence of: subdomainA.example.com with subdomainB.example.com in every text file under the /home/www/ directory tree recursively?
Tedd's user avatar
  • 9,195
854 votes
24 answers
696k views

Bash tool to get nth line from a file

Is there a "canonical" way of doing that? I've been using head -n | tail -1 which does the trick, but I've been wondering if there's a Bash tool that specifically extracts a line (or a range of lines) ...
Vlad Vivdovitch's user avatar
794 votes
31 answers
718k views

Find and kill a process in one line using bash and regex

I often need to kill a process during programming. The way I do it now is: [~]$ ps aux | grep 'python csp_build.py' user 5124 1.0 0.3 214588 13852 pts/4 Sl+ 11:19 0:00 python csp_build.py ...
Orjanp's user avatar
  • 10.7k
655 votes
2 answers
284k views

How to remove double-quotes in jq output for parsing json files in bash?

I'm using jq to parse a JSON file as shown here. However, the results for string values contain the "double-quotes" as expected, as shown below: $ cat json.txt | jq '.name' "Google" How can I pipe ...
Chris F's user avatar
  • 14.8k
581 votes
3 answers
270k views

What is the difference between sed and awk? [closed]

What is the difference between awk and sed ? What kind of application are best use cases for sed and awk tools ?
Rachel's user avatar
  • 101k
449 votes
28 answers
645k views

Using awk to print all columns from the nth to the last

This line worked until I had whitespace in the second field: svn status | grep '\!' | gawk '{print $2;}' > removedProjs Is there a way to have awk print everything in $2 or greater? ($3, $4.. ...
Andy's user avatar
  • 44.9k
388 votes
7 answers
672k views

How do I use shell variables in an awk script?

I found some ways to pass external shell variables to an awk script, but I'm confused about ' and ". First, I tried with a shell script: $ v=123test $ echo $v 123test $ echo "$v" 123test Then tried ...
hqjma's user avatar
  • 3,883
366 votes
9 answers
801k views

How can I use ":" as an AWK field separator?

Given the following command, echo "1: " | awk '/1/ -F ":" {print $1}' why does AWK output: 1: ?
user173446's user avatar
  • 3,719
339 votes
19 answers
207k views

How can I shuffle the lines of a text file on the Unix command line or in a shell script?

I want to shuffle the lines of a text file randomly and create a new file. The file may have several thousands of lines. How can I do that with cat, awk, cut, etc?
Ruggiero Spearman's user avatar
299 votes
7 answers
241k views

AWK: Access captured group from line pattern

If I have an awk command pattern { ... } and pattern uses a capturing group, how can I access the string so captured in the block?
rampion's user avatar
  • 87.6k
291 votes
5 answers
91k views

What are the differences between Perl, Python, AWK and sed? [closed]

What are the main differences among them? And in which typical scenarios is it better to use each language?
Khaled Al Hourani's user avatar
284 votes
8 answers
562k views

Using multiple delimiters in awk

I have a file which contain following lines: /logs/tc0001/tomcat/tomcat7.1/conf/catalina.properties:app.env.server.name = demo.example.com /logs/tc0001/tomcat/tomcat7.2/conf/catalina.properties:app....
Satish's user avatar
  • 16.7k
272 votes
14 answers
317k views

How to show only next line after the matched one?

grep -A1 'blah' logfile Thanks to this command for every line that has 'blah' in it, I get the output of the line that contains 'blah' and the next line that follows in the logfile. It might be a ...
facha's user avatar
  • 12k
260 votes
12 answers
710k views

How to split a delimited string into an array in awk?

How to split the string when it contains pipe symbols | in it. I want to split them to be in array. I tried echo "12:23:11" | awk '{split($0,a,":"); print a[3] a[2] a[1]}' Which works fine. If my ...
Mohamed Saligh's user avatar
251 votes
34 answers
215k views

How can I quickly sum all numbers in a file?

I have a file which contains several thousand numbers, each on its own line: 34 42 11 6 2 99 ... I'm looking to write a script which will print the sum of all numbers in the file. I've got a solution,...
Mark Roberts's user avatar
  • 3,229
243 votes
8 answers
553k views

How to get the second column from command output?

My command's output is something like: 1540 "A B" 6 "C" 119 "D" The first column is always a number, followed by a space, then a double-quoted string. My purpose is to get the second column ...
Qiang Xu's user avatar
  • 4,463
240 votes
3 answers
214k views

How to grep for case insensitive string in a file?

I have a file file1 which ends with Success... OR success... I want to grep for the word success in a way which is not case sensitive way. I have written the following command but it is case sensitive ...
all_techie's user avatar
  • 2,799
222 votes
7 answers
237k views

awk without printing newline

I want the variable sum/NR to be printed side-by-side in each iteration. How do we avoid awk from printing newline in each iteration ? In my code a newline is printed by default in each iteration for ...
Sharat Chandra's user avatar
220 votes
12 answers
287k views

Insert a line at specific line number with sed or awk

I have a script file which I need to modify with another script to insert a text at the 8th line. String to insert: Project_Name=sowstest, into a file called start. I tried to use awk and sed, but ...
ashok's user avatar
  • 2,211
214 votes
10 answers
188k views

Print second-to-last column/field in `awk`

I want to print the second-to-last column or field in awk. The number of fields is the NF variable. I know that I should be able to use $NF, but I'm not sure how it can be used. And this does not seem ...
Brian G's user avatar
  • 53.9k
211 votes
21 answers
315k views

How to merge every two lines into one from the command line?

I have a text file with the following format. The first line is the "KEY" and the second line is the "VALUE". KEY 4048:1736 string 3 KEY 0:1772 string 1 KEY 4192:1349 string 1 KEY 7329:2407 string 2 ...
shantanuo's user avatar
  • 31.7k
207 votes
8 answers
317k views

What is the shortest way to get n-th column of an output?

Let's say that during your workday you repeatedly encounter the following form of columnized output from some command in bash (in my case from executing svn st in my Rails working directory): ? ...
Sv1's user avatar
  • 2,414
201 votes
9 answers
136k views

How to delete duplicate lines in a file without sorting it in Unix

Is there a way to delete duplicate lines in a file in Unix? I can do it with sort -u and uniq commands, but I want to use sed or awk. Is that possible?
Vijay's user avatar
  • 65.7k
197 votes
14 answers
127k views

Sort a text file by line length including spaces

I have a CSV file that looks like this AS2345,ASDF1232, Mr. Plain Example, 110 Binary ave.,Atlantis,RI,12345,(999)123-5555,1.56 AS2345,ASDF1232, Mrs. Plain Example, 1121110 Ternary st. ...
gnarbarian's user avatar
  • 2,632
193 votes
11 answers
335k views

Printing the last column of a line in a file

I have a file that is constantly being written to/updated. I want to find the last line containing a particular word, then print the last column of that line. The file looks something like this. More ...
Rayne's user avatar
  • 14.3k
185 votes
19 answers
79k views

Is there a Unix utility to prepend timestamps to stdin?

I ended up writing a quick little script for this in Python, but I was wondering if there was a utility you could feed text into which would prepend each line with some text -- in my specific case, a ...
Joe Shaw's user avatar
  • 22.2k
181 votes
23 answers
117k views

How can I delete a newline if it is the last character in a file?

I have some files that I'd like to delete the last newline if it is the last character in a file. od -c shows me that the command I run does write the file with a trailing new line: 0013600 n t ...
Todd Partridge 'Gen2ly''s user avatar
172 votes
7 answers
142k views

Save modifications in place with awk

I am learning awk and I would like to know if there is an option to write changes to file, similar to sed where I would use -i option to save modifications to a file. I do understand that I could ...
Deano's user avatar
  • 11.7k
160 votes
10 answers
191k views

How to select lines between two marker patterns which may occur multiple times with awk/sed

Using awk or sed how can I select lines which are occurring between two different marker patterns? There may be multiple sections marked with these patterns. For example: Suppose the file contains: ...
dvai's user avatar
  • 1,963
158 votes
9 answers
610k views

How to print matched regex pattern using awk?

Using awk, I need to find a word in a file that matches a regex pattern. I only want to print the word matched with the pattern. So if in the line, I have: xxx yyy zzz And pattern: /yyy/ I want ...
marverix's user avatar
  • 7,345
155 votes
7 answers
131k views

sed or awk: delete n lines following a pattern

How would I mix patterns and numeric ranges in sed (or any similar tool - awk for example)? What I want to do is match certain lines in a file, and delete the next n lines before proceeding, and I ...
Martin DeMello's user avatar
150 votes
17 answers
184k views

Printing everything except the first field with awk

I have a file that looks like this: AE United Arab Emirates AG Antigua & Barbuda AN Netherlands Antilles AS American Samoa BA Bosnia and Herzegovina BF Burkina Faso BN Brunei Darussalam ...
cfischer's user avatar
  • 24.6k
150 votes
10 answers
219k views

Get last field using awk substr

I am trying to use awk to get the name of a file given the absolute path to the file. For example, when given the input path /home/parent/child/filename I would like to get filename I have tried: ...
Mariappan Subramanian's user avatar
147 votes
9 answers
156k views

How to escape a single quote inside awk

I want do the following awk 'BEGIN {FS=" ";} {printf "'%s' ", $1}' But escaping single quote this way does not work awk 'BEGIN {FS=" ";} {printf "\'%s\' ", $1}' How to do this? Thanks for help.
user avatar
147 votes
3 answers
150k views

What are the differences among grep, awk & sed? [duplicate]

I am confused about the differences between grep, awk and sed in terms of their role in Unix/Linux system administration and text processing.
user288609's user avatar
  • 12.6k
138 votes
10 answers
47k views

How to use multiple arguments for awk with a shebang (i.e. #!)?

I'd like to execute an gawk script with --re-interval using a shebang. The "naive" approach of #!/usr/bin/gawk --re-interval -f ... awk script goes here does not work, since gawk is called with the ...
Dr. Hans-Peter Störr's user avatar
133 votes
6 answers
367k views

using awk with column value conditions

I'm learning awk from The AWK Programming Language and I have a problem with one of the examples. If I wanted to print $3 if $2 is equal to a value (e.g.1), I was using this command which works fine: ...
user1687130's user avatar
  • 1,851
130 votes
10 answers
226k views

How to use awk sort by column 3

I have a file (user.csv)like this ip,hostname,user,group,encryption,aduser,adattr want to print all column sort by user, I tried awk -F ":" '{print|"$3 sort -n"}' user.csv , it doesn't work.
user2452340's user avatar
  • 1,385
128 votes
6 answers
188k views

What are NR and FNR and what does "NR==FNR" imply?

I am learning file comparison using awk. I found syntax like below, awk 'NR==FNR{a[$1];next}$1 in a{print $1}' file1 file2 I couldn't understand what is the significance of NR==FNR in this? If I ...
Amit's user avatar
  • 1,381
126 votes
26 answers
37k views

Is there still any reason to learn AWK?

I am constantly learning new tools, even old fashioned ones, because I like to use the right solution for the problem. Nevertheless, I wonder if there is still any reason to learn some of them. awk ...
Bite code's user avatar
  • 583k
125 votes
19 answers
257k views

Print all but the first three columns

Too cumbersome: awk '{print " "$4" "$5" "$6" "$7" "$8" "$9" "$10" "$11" "$12" "$13}' things
hhh's user avatar
  • 51.2k
123 votes
6 answers
231k views

How to print last two columns using awk

All I want is the last two columns printed.
vehomzzz's user avatar
  • 43.2k
123 votes
1 answer
192k views

Append file contents to the bottom of existing file in Bash [duplicate]

Possible Duplicate: Shell script to append text to each file? How to append output to the end of text file in SHELL Script? I'm trying to work out the best way to insert api details into a pre-...
Grimlockz's user avatar
  • 2,541
119 votes
5 answers
307k views

awk partly string match (if column/word partly matches)

My dummy file looks like this: C1 C2 C3 1 a snow 2 b snowman snow c sowman I want to get line if there is string snow in $3. I can do this like this: awk '($...
pogibas's user avatar
  • 27.6k
118 votes
5 answers
119k views

Show filename and line number in grep output

I am trying to search my rails directory using grep. I am looking for a specific word and I want to grep to print out the file name and line number. Is there a grep flag that will do this for me? I ...
trev9065's user avatar
  • 3,391
114 votes
8 answers
331k views

Tab separated values in awk

How do I select the first column from the TAB separated string? # echo "LOAD_SETTLED LOAD_INIT 2011-01-13 03:50:01" | awk -F'\t' '{print $1}' The above will return the entire line and not ...
shantanuo's user avatar
  • 31.7k
112 votes
4 answers
264k views

Printing column separated by comma using Awk command line

I have a problem here. I have to print a column in a text file using awk. However, the columns are not separated by spaces at all, only using a single comma. Looks something like this: column1,...
user3364728's user avatar
  • 1,147
108 votes
26 answers
127k views

How to decode URL-encoded string in shell?

I have a file with a list of user-agents which are encoded. E.g.: Mozilla%2F5.0%20%28Macintosh%3B%20U%3B%20Intel%20Mac%20OS%20X%2010.6%3B%20en I want a shell script which can read this file and ...
user785717's user avatar
  • 1,285
108 votes
11 answers
119k views

How to print all the columns after a particular number using awk?

On shell, I pipe to awk when I need a particular column. This prints column 9, for example: ... | awk '{print $9}' How can I tell awk to print all the columns including and after column 9, not just ...
Lazer's user avatar
  • 91.4k

1
2 3 4 5
681