2

I am using Mysql 5.1. Because I need transactions I wanted to switch a database from MyISAM to InnoDB, but then I learned I can't because I have a primary key that contains two columns (businessunitID and invoiceID) while one (invoiceID) being an auto increment column:

CREATE TABLE invoice (invoiceID INT auto_increment, businessunitID 
INT(2), date DATETIME, PRIMARY KEY (businessunitID, invoiceID));

This design makes sense, because there are businessunits (like branches) and each has its own set of issued invoices with an ordinal number (invoiceID) where it is not allowed that a number in between is missing.

There are enough answers on stackoverflow that it does not work with InnoDB:

What is wrong with this mysql query?

But my question is: What should I do? Do I really have to migrate very basic database logic to program logic making the application much less failure proof? It is just right to define auto increment at a database level, you'll never know from where the data populating the database could all come from. (Here is an article adressing this: http://weblogs.sqlteam.com/jeffs/archive/2007/08/23/composite_primary_keys.aspx - 100% agree.)

So is there another solution I still can get my desired behaviour with InnoDB?

6
  • To clarify: it sounds like you're expecting auto_increment to produce separate sequences for each distinct value of businessunitID. As in: insert a record for business 1, and it's invoice 1; insert another record for business 1 and it's invoice 2; then insert a record for business 2 and it's invoice 1, not 3. Is that a correct description of what you want?
    – Wyzard
    Apr 2, 2017 at 3:16
  • Wyzard: Yes, that's exactly what it does in MyISAM and it is supposed to do. Works like a charm for 15 years. But now I need transactions.
    – Benedikt
    Apr 2, 2017 at 4:53
  • That seems like a pretty strange and surprising behavior to me. Anyway, auto-generated surrogate keys (i.e. auto_increment) aren't supposed to be meaningful at all. If you have specific business requirements for how your invoices are numbered, you should probably assign the numbers explicitly in your application code.
    – Wyzard
    Apr 2, 2017 at 4:55
  • 1
    Are you trying to generate a sequence of identifiers with no holes? You may need to have an invoice identifier table you can lock to ensure that you don't have duplicates or skip a number.
    – tadman
    Apr 2, 2017 at 5:06
  • 1
    tadman: Yes, that is exactly the case. If I do it in the application code, then I have to care about a lot of things like locking, and yes, nearly the same happens if I do it with triggers and an extra table. The old solution was so cool and easy.
    – Benedikt
    Apr 2, 2017 at 6:49

2 Answers 2

2

I discuss a possible solution for your problem in my blog on converting from MyISAM: https://mariadb.com/kb/en/mariadb/converting-tables-from-myisam-to-innodb/

BEGIN;
SELECT @id := MAX(id)+1 FROM foo WHERE businessunitID = 123 FOR UPDATE;
INSERT INTO foo
    (businessunitID, id, ...)
    VALUES
    (123, @id, ...);
...
COMMIT;

I say "possible solution" because there could be edge cases involving ROLLBACK and crashing that leave gaps.

Please also note that the above code has the same problem that InnoDB has:

  1. Delete the hightest ID
  2. Shutdown the server (gracefully or not)
  3. Startup and allocate an ID -- you will "reuse" the deleted ID. (This is "fixed" in 8.0 for InnoDB, but not in the above code.)

Addenda

  • AUTO_INCREMENT protects itself from multiple connections, etc.
  • Neither engine will ever* duplicate an AUTO_INCREMENT id, regardless of what locking you do or don't do.
  • 2nd column is a feature of MyISAM, unavailable in InnoDB (except by some kludge like my code, above).
  • "Burning ids" (gaps in AUTO_INCREMENT) is possible, in many ways, in both engines.
  • With InnoDB, everything between BEGIN and COMMIT either entirely succeeds or is entirely rolled back; no partial commit (unlike MyISAM). This applies even across power failures.
  • But note... Once you have grabbed @id (in my code), the transactional semantics apply only because of FOR UPDATE. A SELECT with FOR UPDATE announces that you intend to change what was selected -- that is, "locks" it (MAX(ID)).

Footnote: There are two obscure ways to duplicate AUTO_INCREMENT:

  • Dual master, but inserting into both.
  • The delete-shutdown-restart, as mentioned above.

Alternate solution

invoiceID ... AUTO_INCREMENT, 
...
PRIMARY KEY (businessunitID, invoiceID),
INDEX(invoiceID)

AUTO_INCREMENT requires that the id be first in some index, nothing else.

What you lose with this solution, is any hint of the ids being consecutive (1,2,3,...) for a given busineessunit. They will be monotonically increasing.

3
  • I decided to go for your solution. Anyway in your blog you wrote that the transaction will "prevent another thread from grabbing the same id". Is this true? I read a hundred posts about locking / transactions and everybody says something different. Some say the transaction will not prevent another process to grab the same ID as my process while you actually recommend to not use locks with transactions ("LOCK/UNLOCK TABLES — remove them. Replace them (sort of) with BEGIN ... COMMIT.") Or is "FOR UPDATE" that makes the trick and lets transactions behave like locks?
    – Benedikt
    Apr 18, 2017 at 19:37
  • You are mixing things. See Addenda.
    – Rick James
    Apr 18, 2017 at 21:40
  • And another solution.
    – Rick James
    Apr 18, 2017 at 21:44
0

If I had to do this (continue using the MyISAM AUTO_INCREMENT behavior but using InnoDB tables)

One option I would consider is continuing to use MyISAM tables to generate the AUTO_INCREMENT values, exercising that functionality from a BEFORE INSERT trigger on the InnoDB table.

First, I'd remove the AUTO_INCREMENT attribute from the invoiceID column on the new InnoDB table ...

CREATE TABLE invoice
( invoiceID      INT 
, businessunitID INT(2)
, date           DATETIME
, PRIMARY KEY (businessunitID, invoiceID)
) ENGINE=InnoDB;

Next, create a MyISAM table that has the AUTO_INCREMENT property. We only need the two columns in the primary key, we don't need any of the other columns.

CREATE TABLE invoice_autoincrement
( invoiceID      INT AUTO_INCREMENT 
, businessunitID INT(2)
, PRIMARY KEY (businessunitID, invoiceID)
) ENGINE=MyISAM;

Then we can create a BEFORE INSERT trigger. As a rudimentary starting point, something like this:

DELIMITER $$

CREATE TRIGGER invoice_bi
BEFORE INSERT ON invoice 
FOR EACH ROW
BEGIN
   DECLARE li_invoiceid INT DEFAULT NULL;
   IF NEW.invoiceID IS NULL OR NEW.invoiceID = 0 THEN
      INSERT INTO invoice_autoincrement
      ( invoiceID
      , businessunitID
      ) VALUES 
      ( NEW.invoiceID
      , NEW.businessunitID
      );
      SELECT LAST_INSERT_ID() INTO li_invoiceid ;
      SET NEW.invoiceID = li_invoiceid ;
   END IF;
END$$

DELIMITER ;

There's likely more that needs to be added to the trigger to appropriately handle error conditions; if sql_mode doesn't includes NO_AUTO_VALUE_ON_ZERO, then we only need to handle the NULL value and not the zero value...

IF NEW.invoiceID IS NULL THEN 

We also need to consider that the INSERT into the MyISAM table won't be rolled back if the transaction rolls back.

We don't necessarily need to keep every row we insert into the MyISAM table; we only really need the row with the highest (maximum) invoiceID for each businessunitID.

As batch maintenance approach to keeping the MyISAM table trimmed...

DELETE t.*
  FROM invoice_autoincrement t
    -- anti-join exclude 
  LEFT
  JOIN ( SELECT r.businessunitID
              , MAX(r.invoiceid) AS max_invoiceID
           FROM invoice_autoincrement r
          GROUP BY r.businessunitID
       ) s
    ON s.businessunitID = t.businessunitID
   AND s.max_invoiceID  = t.invoiceID
 WHERE s.businessunitID IS NULL

If we want to keep the MyISAM table trimmed from within BEFORE INSERT trigger, following a successful insert to the MyISAM table, and after we've retrieve the LAST_INCREMENT_ID() value... we can execute a DELETE including predicates on businesunitID.

We could repeat the same batch logic, for just the NEW.businessunitID

      DELETE t.*
        FROM invoice_autoincrement t
          -- anti-join exclude 
        LEFT
        JOIN ( SELECT r.businessunitID
                    , MAX(r.invoiceid) AS max_invoiceID
                 FROM invoice_autoincrement r
                WHERE r.businessunitID =   NEW.businessunitID  
                GROUP BY r.businessunitID
             ) s
          ON s.businessunitID = t.businessunitID
         AND s.max_invoiceID  = t.invoiceID
       WHERE s.businessunitID IS NULL
         AND t.businessunitID =    NEW.businessunitID
      ;    

Or we can do a simpler DELETE, removing rows for that one businessunitID that have invoiceID less than the auto_increment value we just retrieved...

      DELETE q.*
        FROM invoice_autoincrement q
       WHERE q.businessunitID = NEW.businessunitID
         AND q.invoiceID      < NEW.invoiceID
      ;

That's an alternative I would consider.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.