Questions tagged [awk]
AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool. AWK is used largely with Unix systems.
32,371
questions
846
votes
36
answers
670k
views
How can I do a recursive find/replace of a string with awk or sed?
How do I find and replace every occurrence of:
subdomainA.example.com
with
subdomainB.example.com
in every text file under the /home/www/ directory tree recursively?
784
votes
19
answers
620k
views
Bash tool to get nth line from a file
Is there a "canonical" way of doing that? I've been using head -n | tail -1 which does the trick, but I've been wondering if there's a Bash tool that specifically extracts a line (or a range of lines) ...
749
votes
29
answers
637k
views
Find and kill a process in one line using bash and regex
I often need to kill a process during programming.
The way I do it now is:
[~]$ ps aux | grep 'python csp_build.py'
user 5124 1.0 0.3 214588 13852 pts/4 Sl+ 11:19 0:00 python csp_build.py
...
553
votes
3
answers
246k
views
What is the difference between sed and awk? [closed]
What is the difference between awk
and sed ?
What kind of application are best use
cases for sed and awk tools ?
463
votes
1
answer
179k
views
How to remove double-quotes in jq output for parsing json files in bash?
I'm using jq to parse a JSON file as shown here. However, the results for string values contain the "double-quotes" as expected, as shown below:
$ cat json.txt | jq '.name'
"Google"
How can I pipe ...
410
votes
27
answers
583k
views
Using awk to print all columns from the nth to the last
This line worked until I had whitespace in the second field.
svn status | grep '\!' | gawk '{print $2;}' > removedProjs
is there a way to have awk print everything in $2 or greater? ($3, $4.. ...
362
votes
9
answers
624k
views
How do I use shell variables in an awk script?
I found some ways to pass external shell variables to an awk script, but I'm confused about ' and ".
First, I tried with a shell script:
$ v=123test
$ echo $v
123test
$ echo "$v"
123test
Then tried ...
322
votes
9
answers
688k
views
How can I use ":" as an AWK field separator?
Given the following command,
echo "1: " | awk '/1/ -F ":" {print $1}'
why does AWK output:
1:
?
321
votes
19
answers
196k
views
How can I shuffle the lines of a text file on the Unix command line or in a shell script?
I want to shuffle the lines of a text file randomly and create a new file. The file may have several thousands of lines.
How can I do that with cat, awk, cut, etc?
274
votes
5
answers
86k
views
What are the differences between Perl, Python, AWK and sed? [closed]
What are the main differences among them? And in which typical scenarios is it better to use each language?
274
votes
7
answers
212k
views
AWK: Access captured group from line pattern
If I have an awk command
pattern { ... }
and pattern uses a capturing group, how can I access the string so captured in the block?
258
votes
13
answers
293k
views
How to show only next line after the matched one?
grep -A1 'blah' logfile
Thanks to this command for every line that has 'blah' in it, I get the output of the line that contains 'blah' and the next line that follows in the logfile. It might be a ...
258
votes
8
answers
493k
views
Using multiple delimiters in awk
I have a file which contain following lines:
/logs/tc0001/tomcat/tomcat7.1/conf/catalina.properties:app.env.server.name = demo.example.com
/logs/tc0001/tomcat/tomcat7.2/conf/catalina.properties:app....
231
votes
12
answers
606k
views
How to split a delimited string into an array in awk?
How to split the string when it contains pipe symbols | in it.
I want to split them to be in array.
I tried
echo "12:23:11" | awk '{split($0,a,":"); print a[3] a[2] a[1]}'
Which works fine. If my ...
224
votes
32
answers
188k
views
How can I quickly sum all numbers in a file?
I have a file which contains several thousand numbers, each on it's own line:
34
42
11
6
2
99
...
I'm looking to write a script which will print the sum of all numbers in the file. I've got a ...
207
votes
8
answers
449k
views
How to get the second column from command output?
My command's output is something like:
1540 "A B"
6 "C"
119 "D"
The first column is always a number, followed by a space, then a double-quoted string.
My purpose is to get the second column ...
206
votes
6
answers
214k
views
awk without printing newline
I want the variable sum/NR to be printed side-by-side in each iteration. How do we avoid awk from printing newline in each iteration ? In my code a newline is printed by default in each iteration
for ...
204
votes
8
answers
314k
views
bash: shortest way to get n-th column of output
Let's say that during your workday you repeatedly encounter the following form of columnized output from some command in bash (in my case from executing svn st in my Rails working directory):
? ...
194
votes
9
answers
169k
views
Print second-to-last column/field in `awk`
I want to print the second-to-last column or field in awk. The number of fields is the NF variable. I know that I should be able to use $NF, but I'm not sure how it can be used.
And this does not seem ...
193
votes
11
answers
248k
views
Insert a line at specific line number with sed or awk
I have a script file which I need to modify with another script to insert a text at the 8th line.
String to insert: Project_Name=sowstest, into a file called start.
I tried to use awk and sed, but ...
191
votes
2
answers
155k
views
How to grep for case insensitive string in a file?
I have a file file1 which ends with
Success... OR
success...
I want to grep for the word success in a way which is not case sensitive way.
I have written the following command but it is case ...
190
votes
21
answers
285k
views
How to merge every two lines into one from the command line?
I have a text file with the following format. The first line is the "KEY" and the second line is the "VALUE".
KEY 4048:1736 string
3
KEY 0:1772 string
1
KEY 4192:1349 string
1
KEY 7329:2407 string
2
...
182
votes
11
answers
306k
views
Printing the last column of a line in a file
I have a file that is constantly being written to/updated. I want to find the last line containing a particular word, then print the last column of that line.
The file looks something like this. More ...
180
votes
19
answers
77k
views
Is there a Unix utility to prepend timestamps to stdin?
I ended up writing a quick little script for this in Python, but I was wondering if there was a utility you could feed text into which would prepend each line with some text -- in my specific case, a ...
180
votes
9
answers
122k
views
How to delete duplicate lines in a file without sorting it in Unix
Is there a way to delete duplicate lines in a file in Unix?
I can do it with sort -u and uniq commands, but I want to use sed or awk.
Is that possible?
179
votes
23
answers
113k
views
How can I delete a newline if it is the last character in a file?
I have some files that I'd like to delete the last newline if it is the last character in a file. od -c shows me that the command I run does write the file with a trailing new line:
0013600 n t ...
179
votes
13
answers
108k
views
Sort a text file by line length including spaces
I have a CSV file that looks like this
AS2345,ASDF1232, Mr. Plain Example, 110 Binary ave.,Atlantis,RI,12345,(999)123-5555,1.56
AS2345,ASDF1232, Mrs. Plain Example, 1121110 Ternary st. ...
158
votes
7
answers
127k
views
Save modifications in place with awk
I am learning awk and I would like to know if there is an option to write changes to file, similar to sed where I would use -i option to save modifications to a file.
I do understand that I could ...
146
votes
10
answers
175k
views
How to select lines between two marker patterns which may occur multiple times with awk/sed
Using awk or sed how can I select lines which are occurring between two different marker patterns? There may be multiple sections marked with these patterns.
For example:
Suppose the file contains:
...
145
votes
3
answers
146k
views
What are the differences among grep, awk & sed? [duplicate]
Possible Duplicate:
What are the differences between Perl, Python, AWK and sed?
What is the difference between sed and awk?
Maybe not a very specific question, but I am confused about the ...
142
votes
9
answers
545k
views
How to print matched regex pattern using awk?
Using awk, I need to find a word in a file that matches a regex pattern.
I only want to print the word matched with the pattern.
So if in the line, I have:
xxx yyy zzz
And pattern:
/yyy/
I want ...
140
votes
6
answers
116k
views
sed or awk: delete n lines following a pattern
How would I mix patterns and numeric ranges in sed (or any similar tool - awk for example)? What I want to do is match certain lines in a file, and delete the next n lines before proceeding, and I ...
135
votes
16
answers
159k
views
Printing everything except the first field with awk
I have a file that looks like this:
AE United Arab Emirates
AG Antigua & Barbuda
AN Netherlands Antilles
AS American Samoa
BA Bosnia and Herzegovina
BF Burkina Faso
BN Brunei Darussalam
...
135
votes
7
answers
142k
views
How to escape a single quote inside awk
I want do the following
awk 'BEGIN {FS=" ";} {printf "'%s' ", $1}'
But escaping single quote this way does not work
awk 'BEGIN {FS=" ";} {printf "\'%s\' ", $1}'
How to do this? Thanks for help.
134
votes
10
answers
188k
views
Get last field using awk substr
I am trying to use awk to get the name of a file given the absolute path to the file.
For example, when given the input path /home/parent/child/filename I would like to get filename
I have tried:
...
131
votes
10
answers
43k
views
How to use multiple arguments for awk with a shebang (i.e. #!)?
I'd like to execute an gawk script with --re-interval using a shebang. The "naive" approach of
#!/usr/bin/gawk --re-interval -f
... awk script goes here
does not work, since gawk is called with the ...
126
votes
6
answers
332k
views
using awk with column value conditions
I'm learning awk from The AWK Programming Language and I have a problem with one of the examples.
If I wanted to print $3 if $2 is equal to a value (e.g.1), I was using this command which works fine:
...
124
votes
19
answers
253k
views
Print all but the first three columns
Too cumbersome:
awk '{print " "$4" "$5" "$6" "$7" "$8" "$9" "$10" "$11" "$12" "$13}' things
120
votes
25
answers
33k
views
Is there still any reason to learn AWK?
I am constantly learning new tools, even old fashioned ones, because I like to use the right solution for the problem.
Nevertheless, I wonder if there is still any reason to learn some of them. awk ...
118
votes
6
answers
215k
views
How to print last two columns using awk
All I want is the last two columns printed.
114
votes
1
answer
184k
views
Append file contents to the bottom of existing file in Bash [duplicate]
Possible Duplicate:
Shell script to append text to each file?
How to append output to the end of text file in SHELL Script?
I'm trying to work out the best way to insert api details into a pre-...
112
votes
6
answers
150k
views
What are NR and FNR and what does "NR==FNR" imply?
I am learning file comparison using awk.
I found syntax like below,
awk 'NR==FNR{a[$1];next}$1 in a{print $1}' file1 file2
I couldn't understand what is the significance of NR==FNR in this?
If I ...
110
votes
9
answers
193k
views
How to use awk sort by column 3
I have a file (user.csv)like this
ip,hostname,user,group,encryption,aduser,adattr
want to print all column sort by user,
I tried awk -F ":" '{print|"$3 sort -n"}' user.csv , it doesn't work.
108
votes
5
answers
104k
views
Show filename and line number in grep output
I am trying to search my rails directory using grep. I am looking for a specific word and I want to grep to print out the file name and line number.
Is there a grep flag that will do this for me? I ...
107
votes
5
answers
84k
views
Using awk to remove the Byte-order mark
How would an awk script (presumably a one-liner) for removing a BOM look like?
Specification:
print every line after the first (NR > 1)
for the first line: If it starts with #FE #FF or #FF #FE, ...
106
votes
4
answers
227k
views
Printing column separated by comma using Awk command line
I have a problem here. I have to print a column in a text file using awk. However, the columns are not separated by spaces at all, only using a single comma. Looks something like this:
column1,...
103
votes
6
answers
301k
views
Tab separated values in awk
How do I select the first column from the TAB separated string?
# echo "LOAD_SETTLED LOAD_INIT 2011-01-13 03:50:01" | awk -F'\t' '{print $1}'
The above will return the entire line and not ...
102
votes
5
answers
273k
views
awk partly string match (if column/word partly matches)
My dummy file looks like this:
C1 C2 C3
1 a snow
2 b snowman
snow c sowman
I want to get line if there is string snow in $3. I can do this like this:
awk '($...
102
votes
13
answers
91k
views
how to use sed, awk, or gawk to print only what is matched?
I see lots of examples and man pages on how to do things like search-and-replace using sed, awk, or gawk.
But in my case, I have a regular expression that I want to run against a text file to extract ...