40

I have to merge multiple CSV files with same headers. I have to keep the header of the first file and remove headers of all the other files and merge them and create one master file.

file 1:

Id,city,name ,location
1,NA,JACK,CA

file 2:

ID,city,name,location
2,NY,JERRY,NY

output:

Id,city,name,location
1,NA,JACK,CA
2,NY,JERRY,NY

Currently I am using this code:

ls *.csv | xargs -n 1 tail -n+2 > master.csv

This code will merge the files perfectly , but as I need the header of the first file, this will not give me the header.

What should I do?

1
  • This is an append operation, not merge. Perhaps someone can edit the question title to better represent the content.
    – Lucas
    Mar 4, 2021 at 21:42

4 Answers 4

88
awk 'FNR==1 && NR!=1{next;}{print}' *.csv

tested on solaris unix:

> cat file1.csv
Id,city,name ,location
1,NA,JACK,CA
>
> cat file2.csv
ID,city,name,location
2,NY,JERRY,NY
>
> nawk 'FNR==1 && NR!=1{next;}{print}' *.csv
Id,city,name ,location
1,NA,JACK,CA
2,NY,JERRY,NY
> 

Explanation given by kevin-d:

FNR is the number of lines (records) read so far in the current file. NR is the number of lines read overall. So the condition 'FNR==1 && NR!=1{next;}' says, "Skip this line if it's the first line of the current file, and at least 1 line has been read overall." This has the effect of printing the CSV header of the first file while skipping it in the rest.

Link for the difference between and

2
  • 10
    Explanation: FNR is the number of lines (records) read so far in the current file. NR is the number of lines read overall. So the condition 'FNR==1 && NR!=1{next;}' says, "Skip this line if it's the first line of the current file, and at least 1 line has been read overall." This has the effect of printing the CSV header of the first file while skipping it in the rest.
    – Kevin D.
    Jul 9, 2014 at 19:59
  • 10
    For the noobs like me... I added the > master.csv to the end to create the new file with all the data instead of it running the the CLI Apr 7, 2015 at 7:24
2

If Perl is an option:

perl -ne 'print if $. > 1 or ! $h; $h=1; close ARGV if eof' *.csv > master.csv

$. is the line number.
It is NOT reset automatically between files, so close ARGV if eof is needed.
$h records if the header has already been printed.

0

Just as a side note for everyone who uses the accepted solution of this thread (like me as well :)) - be careful that this code will fail if the header contains new lines, i.e., something like

column1,"column\nwith\new line",column2
value1,value2,value3
...

In this case, only the part column1,"column will be considered as the header and the rest of the header will be considered a normal row (which will completely break your final CSV). If you have a header with a new line inside, the only solution I can think about is to use a "full-fledged" csv reader library which will be able to correctly read the header.

But despite of this minor issue, the line above saved me from a lot of head ache. :D

-1
<?php
ini_set('auto_detect_line_endings', true);
$dir = "include/*.csv";
$returnVal = array();
foreach (glob($dir) as $file) {
    $header = null;
    $file = fopen($file, 'r') or die('Unable to open file!');
    while(($row = fgetcsv($file)) !== false){
        if($header === null){
            $header = $row;
            continue;
        }
        $newRow = array();
        for($i = 0; $i<count($row); $i++){

            $newRow[] = $row[$i];   
        }
        if($newRow[0] == null)
        break;
        else
        $returnVal[] = $newRow;
    }
    fclose($file);
}
//var_dump($returnVal);
$output = fopen("file.csv",'w') or die("Can't open output");
fputcsv($output, array('Date','close','open'));
foreach($returnVal as $product) {
    fputcsv($output, $product);
}

fclose($output) or die("Can't close php://output"); ?>

1
  • 3
    Pasting code is not enough. Tell us something about it.
    – kwoxer
    Mar 8, 2017 at 11:38

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.