10

We have a need to compare two CSV files. Let say file one have a few rows, and second file could have the same no of rows or more. Most of the rows could remain same on both files.Looking for the best approach to do a diff between these two files and read only those rows which has a difference in the second file from the first file. The application processing the file is in Java.

What are the best approaches for this?

Note : it would be great if we can know a row is updated, inserted or deleted in the second file.

Requirements:-

  1. There won't be any duplicate records
  2. File 1 and file 2 could have same no of records with a few rows with updated values in file2 (Records updated)
  3. File 2 could have a few rows removed ( this is treated as record deleted)
  4. File 2 could have a few new rows added ( this is treated as record inserted)
  5. On of the column could be treated a the primary key of the record, that won't change in both the files.
5
  • 1
    Can the files contain duplicate rows? Do they files have the same columns or are the columns of one file a subset of the columns of the other one?
    – G. Bach
    Jun 2, 2012 at 18:06
  • Duplicates are not possible. The order many not be same. The second file could have same no of rows with no update, a few rows updated, a few row deleted or a few new rows added. We would need to pick up all these changes by comparing both the files.
    – Java Guy
    Jun 2, 2012 at 18:08
  • How would you define "updated"? Is there an id? It maybe would have been good to specify these assumptions in the question from the get-go.
    – beerbajay
    Jun 2, 2012 at 18:14
  • I am looking for something similar from this post stackoverflow.com/questions/9528202/…
    – Java Guy
    Jun 2, 2012 at 18:16
  • Uh, that's still not really clear, can you edit your question to be more specific?
    – beerbajay
    Jun 2, 2012 at 18:19

7 Answers 7

11

One method for doing this would be to use java's Set interface; read each line as a string, add it to the set, then do a removeAll() with the second set on the first set, thus retaining the rows which differ. This, of course, assumes that there are no duplicate rows in the files.

// using FileUtils to read in the files.
HashSet<String> f1 = new HashSet<String>(FileUtils.readLines("file1.csv"));
HashSet<String> f2 = new HashSet<String>(FileUtils.readLines("file2.csv"));
f1.removeAll(f2); // f1 now contains only the lines which are not in f2

Update

Okay, so you have a PK field. I'll just assume you know how to get that from your string; use openCSV or regex or whatever you want. Make an actual HashMap instead of a HashSet as above, use the PK as the key and the row as the value.

HashMap<String, String> f1 = new HashMap<String, String>();
HashMap<String, String> f2 = new HashMap<String, String>();
// read f1, f2; use PK field as the key
List<String> deleted = new ArrayList<String>();
List<String> updated = new ArrayList<String>();
for(Map.Entry<String, String> entry : f1.keySet()) {
    if(!f2.containsKey(entry.getKey()) {
        deleted.add(entry.getValue());
    } else {
        if(!f2.get(entry.getKey().equals(f1.getValue())) {
            updated.add(f1.getValue());
        }
    }
}
for(String key : f1.keySet()) {
    f2.remove(key);
}
// f2 now contains only "new" rows
9
  • That's a pretty good solution. But what if there are duplicates in each file?
    – user377628
    Jun 2, 2012 at 18:06
  • 1
    Well, if there are duplicates in f1 they'll be 'collapsed' into a single entry in the set. It will only take a single "same" row in f2 for that entry to be removed from f1. This may or may not be an issue for the OP.
    – beerbajay
    Jun 2, 2012 at 18:08
  • Well its a good solution but the issue mentioned above doesn't handle the duplicates in a single file. Plus it also does not take care of the situation where an extra space shows up after the comma and it screws things up.. I am talking specially about the space character which has ascii code 32. Most of the times in csv and other text files the space is abused.
    – dharam
    Jun 2, 2012 at 18:09
  • I just saw a comment above, and the OP said that duplicates are not possible within a file, so you're good there.
    – user377628
    Jun 2, 2012 at 18:10
  • 1
    @dharam That is certainly true. The OP didn't specify more than "same rows". If they said "my data is really filthy", then they should either (a) clean up the data first (b) read it into a class and implement compareTo(), then use the same set method as above. Our answers are not really intended to be general purpose solutions; they're supposed to be direct answers to the question as stated.
    – beerbajay
    Jun 2, 2012 at 18:11
5

Read the entire first file, and put it into a List. Then read the second file one row at a time, and compare each row to all the rows of the first file to see if it's a duplicate. If it's not a duplicate, then it's new information. If you're having trouble with reading, look at http://opencsv.sourceforge.net/, it's a pretty good library for reading CSV files in Java.

1
  • thanks, Hassan. Yes, opencsv is good option to read and compare. I am still looking for a way we can get the difference on the fly.
    – Java Guy
    Jun 2, 2012 at 18:07
3

Try using the java-diff-utils library

Example

I use groovy for quick demos of java libraries:

The following differences are reported between two sample files:

$ groovy diff
[ChangeDelta, position: 0, lines: [1,11,21,31,41,51] to [1,11,99,31,41,51]]
[DeleteDelta, position: 2, lines: [3,13,23,33,43,53]]
[InsertDelta, position: 5, lines: [6,16,26,36,46,56]]

files1.csv

1,11,21,31,41,51
2,12,22,32,42,52
3,13,23,33,43,53
4,14,24,34,44,54
5,15,25,35,45,55

file2.csv

1,11,99,31,41,51
2,12,22,32,42,52
4,14,24,34,44,54
5,15,25,35,45,55
6,16,26,36,46,56

diff.groovy

//
// Dependencies
// ============
import difflib.*

@Grapes([
    @Grab(group='com.googlecode.java-diff-utils', module='diffutils', version='1.2.1'),
])

//
// Main program
// ============
def original = new File("file1.csv").readLines()
def revised  = new File("file2.csv").readLines()

Patch patch = DiffUtils.diff(original, revised)

patch.getDeltas().each {
    println it
}

Update

According to the dbunit FAQ performance of this solution can be improved for very large datasets by using a streamed revision of the ResultSetTableFactory interface. This is enabled within the ANT task as follows:

ant.dbunit(driver:driver, url:url, userid:user, password:pass) {
    compare(src:"dbunit.xml", format:"flat")
    dbconfig {
        property(name:"datatypeFactory", value:"org.dbunit.ext.h2.H2DataTypeFactory")
        property(name:"resultSetTableFactory", value:"org.dbunit.database.ForwardOnlyResultSetTableFactory")
    }
}
2
  • Thanks Mark. Just concerned if file grows bigger there will be issue that we need more memory.
    – Java Guy
    Jun 4, 2012 at 23:09
  • @JavaGuy The dbunit guys have thought of this. Checkout the following FAQ section: dbunit.org/faq.html#streaming Jun 5, 2012 at 17:21
2

There is a program that compare/subtract the two CSV files. It uses the ArrayList

import java.io.*;
import java.util.ArrayList;

/* file1 - file2 = file3*/
public class CompareCSV {
public static void main(String args[]) throws FileNotFoundException, IOException
{
    String path="D:\\csv\\";
    String file1="file1.csv";
    String file2="file2.csv";
    String file3="p3lang.csv";
    ArrayList al1=new ArrayList();
    ArrayList al2=new ArrayList();
    //ArrayList al3=new ArrayList();

    BufferedReader CSVFile1 = new BufferedReader(new FileReader(path+file1));
    String dataRow1 = CSVFile1.readLine();
    while (dataRow1 != null)
    {
        String[] dataArray1 = dataRow1.split(",");
        for (String item1:dataArray1)
        { 
           al1.add(item1);
        }

        dataRow1 = CSVFile1.readLine(); // Read next line of data.
    }

     CSVFile1.close();

    BufferedReader CSVFile2 = new BufferedReader(new FileReader(path+file2));
    String dataRow2 = CSVFile2.readLine();
    while (dataRow2 != null)
    {
        String[] dataArray2 = dataRow2.split(",");
        for (String item2:dataArray2)
        { 
           al2.add(item2);

        }
        dataRow2 = CSVFile2.readLine(); // Read next line of data.
    }
     CSVFile2.close();

     for(String bs:al2)
     {
         al1.remove(bs);
     }

     int size=al1.size();
     System.out.println(size);

     try
        {
            FileWriter writer=new FileWriter(path+file3);
            while(size!=0)
            {
                size--;
                writer.append(""+al1.get(size));
                writer.append('\n');
            }
            writer.flush();
            writer.close();
        }
        catch(IOException e)
        {
            e.printStackTrace();
        }
}}

http://p3lang.com/subtract-one-csv-from-another-in-java/

0
0

you mentioned detecting "updated" rows. I guess this implies that a row has an identity in some way that survives the update. Maybe a single column, or a compound column provides identity. Thats an implementation detail that you personally need to sort out and implement, and it will only add more code to your solution.

Anyway...databases tend to have good support for working with set data, and loading data from csv files. All the big name relational databases have great support with easy syntax for loading the data in a csv file into a table. At that point, finding new rows, or modified rows between two tables are very simple sql queries.

its obviously not a pure java solution, but worth mentioning i think.

0

My simple solution in case you want to compare two csv responses stored in string variables (in the case you get them through a REST call). In my case I wanted to exit the check after a threshold of 10 different lines.

        BufferedReader baseline = new BufferedReader(new StringReader(responseBaseline));
        BufferedReader tested = new BufferedReader(new StringReader(responseTested));
        String lineBaseline = null;
        String lineTested = null;
        boolean linesExist = true;
        boolean foundDiff = false;
        int lineNumber = 0;
        int errorNumber = 0;
        int errorThreshold = 10;
        String message = "";
        while (linesExist) {
            try {
                lineBaseline = baseline.readLine();
                lineTested = tested.readLine();
                lineNumber++;
                if ((lineBaseline != null) && (lineTested != null)) {
                    if (!lineTested.equals(lineBaseline)) {
                        foundDiff = true;
                        errorNumber++;
                        if (errorNumber > errorThreshold) {
                            message = message + "\r\n" + "Found more than " + errorThreshold + " lines that were different. Will exit check.";
                            break;
                        }
                        message = message + "\r\n" + "\r\n#Found differences for line number " + lineNumber + "\r\nLine baseline: " + lineBaseline + "\r\nLine tested: " + lineTested;
                    }
                } else {
                    linesExist = false;
                }
            } catch (IOException e) {
                throw new Error("Problems with reading csv files");
            }
        }
        if (foundDiff) {
            throw new Error("Found differences between csv files. " + message);
        }
    }
-1

What I suggest:

You can read a file to create tokens separated by , and trim each token from both the sides so that extra spaces are taken care of and then store them in a data structure which is ordered ( similar to linked hash set , linked hash map etc ( if you want to get pass the duplicates in a file in case if there is any) and then repeat it for the other file.

Java provides many utility methods to compare these data structures. :)

Not the answer you're looking for? Browse other questions tagged or ask your own question.