31

What is the best way to find records with duplicate values in a column using ruby and the new Activerecord?

2
  • Duplicated values of just 1 column or more than 1? Is/Are these string/ints/text fields? Commented Feb 24, 2011 at 14:54
  • just 1 column - strings.
    – srboisvert
    Commented Feb 24, 2011 at 15:23

6 Answers 6

51

Translating @TuteC into ActiveRecord:

sql = 'SELECT id, 
         COUNT(id) as quantity 
         FROM types 
         GROUP BY name 
       HAVING quantity > 1'
#=>
Type.select("id, count(id) as quantity")
  .group(:name)
  .having("quantity > 1")
8
  • 7
    Is this code supposed to work for PostgreSQL as well? It returns the error PGError: ERROR: column "quantity" does not exist
    – Marc
    Commented Jan 29, 2012 at 2:33
  • 6
    @Marc, I am not sure. But you can try Type.select("id, count(id) as quantity").group(:name).having("count(id) > 1")
    – fl00r
    Commented Jan 29, 2012 at 10:47
  • this will only find exact duplicates.
    – OpenCoderX
    Commented Dec 21, 2012 at 15:04
  • 3
    @holaSenor - What is a non-exact duplicate, exactly? (pun intended) Commented Nov 19, 2014 at 19:34
  • 1
    re fl00rs answer: Postgresql - ActiveRecord::StatementInvalid: PG::GroupingError: ERROR: column "types.id" must appear in the GROUP BY clause or be used in an aggregate function. But add it to the group-by (no error), and then you get zero hits, since no two records have the same id-value. I've tried several of these solutions, and all either don't return the ID in the result (so who knows where the dups were), or fail to find the desired records when the ID is included.
    – JosephK
    Commented Nov 26, 2017 at 12:23
28

Here's how I solved it with the AREL helpers, and no custom SQL:

Person.select("COUNT(last_name) as total, last_name")
  .group(:last_name)
  .having("COUNT(last_name) > 1")
  .order(:last_name)
  .map{|p| {p.last_name => p.total} }

Really, it's just a nicer way to write the SQL. This finds all records that have duplicate last_name values, and tells you how many and what the last names are in a nice hash.

0
21

I was beating my head against this problem with a 2016 stack (Rails 4.2, Ruby 2.2), and got what I wanted with this:

> Model.select([:thing]).group(:thing).having("count(thing) > 1").all.size
 => {"name1"=>5, "name2"=>4, "name3"=>3, "name4"=>2, "name5"=>2}
1
  • 3
    this was exactly what I needed Commented Sep 21, 2017 at 13:35
11

With custom SQL, this finds types with same values for name:

sql = 'SELECT id, COUNT(id) as quantity FROM types
         GROUP BY name HAVING quantity > 1'
repeated = ActiveRecord::Base.connection.execute(sql)
6

In Rails 2.x, select is a private method of AR class. Just use find():

klass.find(:all, 
  :select => "id, count(the_col) as num", 
  :conditions => ["extra conditions here"], 
  :group => 'the_col', 
  :having => "num > 1")
2
  • 3
    Totally off-topic, grave-digging comment here... but WHY IS THIS BETTER THAN SQL? Commented Dec 21, 2012 at 17:33
  • @JohnCromartie It's not. But it's fun. ;)
    – Nowaker
    Commented Dec 10, 2013 at 16:34
2

Here is a solution that extends the other answers to show how to find and iterate through the records grouped by the duplicate field:

duplicate_values = Model.group(:field).having(Model.arel_table[:field].count.gt(1)).count.keys
Model.where(field: duplicate_values).group_by(&:field).each do |value, records|
  puts "The records with ids #{records.map(&:id).to_sentence} have field set to #{value}"
end

It seems a shame this has to be done with two queries but this answer confirms this approach.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.