45

I have a table with 3 columns - id (pk), pageId (fk), name. I have a php script which dumps about 5000 records into the table, with about half being duplicates, with same pageId and name. Combination of pageId and name should be unique. What is the best way to prevent duplicates being saved to the table as I loop through the script in php?

4 Answers 4

126

First step would be to set a unique key on the table:

ALTER TABLE thetable ADD UNIQUE INDEX(pageid, name);

Then you have to decide what you want to do when there's a duplicate. Should you:

  1. ignore it?

    INSERT IGNORE INTO thetable (pageid, name) VALUES (1, "foo"), (1, "foo");
    
  2. Overwrite the previously entered record?

    INSERT INTO thetable (pageid, name, somefield)
    VALUES (1, "foo", "first")
    ON DUPLICATE KEY UPDATE (somefield = 'first')
    
    INSERT INTO thetable (pageid, name, somefield)
    VALUES (1, "foo", "second")
    ON DUPLICATE KEY UPDATE (somefield = 'second')
    
  3. Update some counter?

    INSERT INTO thetable (pageid, name)
    VALUES (1, "foo"), (1, "foo")
    ON DUPLICATE KEY UPDATE (pagecount = pagecount + 1)
    
1
  • 6
    BIG thanks, this was an excellent answer to a question I researched for some time here on Stack Overlow.
    – capfu
    Commented Oct 25, 2011 at 6:29
5

You can also ignore the error with mysql: INSERT IGNORE INTO TABLE ... it will ignore the key error, skip over that insert and move on to the next.

4

From a mysql point you can do

alter table YOURTABLE add unique index(pageId, name);

If your wording is correct and you want to do it from php you can do

$already_done = array();
foreach ($records as $record)
{
   $unique_hash = md5($record['name'].$record['pageId']);
   if (!in_array($unique_hash, $already_done))
   {
      $already_done[] = $unique_hash;
      // sql insert here
   }
}

either way those should do you just fine.

1
  • 1
    Of course, if there are already records in the table before the script starts, those wouldn't appear in $already_done. Commented Feb 8, 2010 at 7:00
0

You can set the PageID and Name to a Unique index in the MySQL database. This way when you insert the rows, it will cause an error, which can be ignored by PHP, and you can just go to the next row.

This assumes you are inserting rows individually. AKA:

foreach($large_data as $fields)
{
    mysql_query("INSERT INTO TABLE (`Something`) VALUES('".$fields['something']."');
}
1
  • 2
    Intentionally allowing mysql_query() to throw PHP warnings when you hit duplicate rows is kind of messy (it clutters your log, it's relatively resource intensive, etc.). Especially when preventing them in the first place is as simple as using MySQL's INSERT IGNORE feature mentioned in other solutions. Additionally, inserting in a loop is inefficient when bulk insert syntax is available. Commented Feb 8, 2010 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.