1

I realize I am asking two separate questions at once here but I think they are related (even if just slightly). Anyway what I want to do is compare two lists (not necessarily Java lists) of Strings and remove words that occur in both lists. I was thinking of using either an ArrayList or a HashSet withHashSet being favored as the lists are not ordered but my problem with a HashSet is that I've read that they don't allow duplicates. This conflicts with my other requirement slightly as I want to be able to count the amount of times each word occurs but only display them once...if that makes sense. Think of a WordCloud example.

Here's what I have currently, saving contents of two text files to two ArrayLists:

ArrayList<String> words = new ArrayList<String>();
        File file = new File(fileName);
        Scanner scanner = new Scanner(file).useDelimiter("$");
        while(scanner.hasNext())
        {               
            String wrd = scanner.nextLine();
            words.add(wrd);
        }

I had to use two different ways of saving data as the two text files were structured differently

ArrayList<String> webWords = new ArrayList<String>();
    File webFile = new File(webFileName);
    BufferedReader br = new BufferedReader(new FileReader(webFileName));
    String testLine = "", str = "";
    int count = 0;
    String s;
    while ((testLine = br.readLine()) != null) {
            str += testLine + " ";
    }
    StringTokenizer st = new StringTokenizer(str);
    while (st.hasMoreTokens()) {
            s = st.nextToken();
            webWords.add(s);
            count++;
    }

Now I could easily create two HashSets in a similar fashion but I am using ArrayList for the moment as it allows duplicates and I am still unsure on which suits my needs best.

I need to compare the 2nd list with the 1st one and remove all the words in the 2nd list that appear in the 1st list.

My second issue is trying to determine (after I've removed the common words) which words occur most frequently.

Any help or direction would be greatly appreciated.

3 Answers 3

3

If I understand the requierements correctly then we can take a HashMap<String, Integer> and put all words from list1 in it as keys, thus we avoid duplicates

for(String w : list1) {
      map.put(w, 0);
}

then we can iterate over the map entries, count word frequency and put it as entry value

for(Entry<String, Integer> e : map.entrySet) {
      int n = Collections.frequency(list2, e.getKey());
      e.setValue(n);
}

UPDATE: "I want to be able to remove the words from list2 that appear in list1. And then iterate through the remaining words in list2 to find out how many times each word occurs"

   list2.removeAll(list1);

   for(String w : list2) {
          map.put(w, 0);
    }

    for(Entry<String, Integer> e : map.entrySet) {
          int n = Collections.frequency(list2, e.getKey());
          e.setValue(n);
    }
2
  • Thank you, Evgeniy! However, from what I can see this code checks how many times the words in list1 occur in list 2. This is fine but what I want to be able to do is remove the words from list2 that appear in list1. And then iterate through the remaining words in list2 to find out how many times each word occurs. Jan 9, 2013 at 18:37
  • Thank you Evgeniy! Much appreciated. Now I can maybe Order by Value to display most frequent words. Thanks again, I had been trying similar efforts as the ones below but this suits my needs better. Jan 9, 2013 at 19:16
1

To remove all elements of one collection that are present in another, use removeAll. This is an optional operation in the Collection interface, and both ArrayList and HashSet implement it.

words.removeAll(webWords);

To build a frequency table, you must not be using a Set, since Sets only keep one of each element. You'll want to use a List and create a Map<String, Integer> to map each word to the number of times it appears. To build it, you can do something like this:

for (String word : words) {
    if (freqMap.containsKey(word)) {
        freqMap.put(word, freqMap.get(word) + 1); // increment
    } else {
        freqMap.put(word, 1); // begin at 1 if it was not present
    }
}
0

The simple way to do a combination of terms and counts is to use a HashMap with the word as the key and the number of occurrences as the value. I don't recall the exact notation for Java , but I would expect something like this:

HashMap words = new HashMap();
while ( st.hasMoreTokens())
{
   s = st.nextToken();
   if ( words.containsKey(s) )
    {
        // probably not how indexing is done, but would be in most languages :p
        words[s]= words[s]+1;
     }
   else 
     {
       words.put( s, 1 );
     }
}

That way you end up with a HashMap of all unique words and a count of appearances for each one.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.