1

I am currently running the following stored procedure. While it is a lot more efficient than my original procedure it is still taking an excessive amount of time. I'm not actually sure what the slow down is as the first 10k-30k records happened fast, but it has grown slower and slower as it gets further in. I'm expecting to update about 250k rows of about 1.7 million. Once this is complete I will then be doing something similar to insert records into each "Solar System".

To give you an example of the time this is taking. It has now been running for a little over 24 hours and it is only on iteration 786 of the 1716 it has to do. The reason for the changing limit on the selects is that there are 1000 possible rows per a sector in my table. I don't personally see any slow downs, but then I don't understand the inner workings of MySQL that well.

This work is being done on my local computer, no it is not slow, but there is always the possibility that there are changes that need to be done at the server level that would make these queries more efficient. If need be I can change the server settings so that is a possibility also. FYI I'm using the stock configuration from MySQL on a Windows 7.

DECLARE CurrentOffset int; -- Current offset limit to only deal with one 
DECLARE CurrentOffsetMultiplier int;
DECLARE RandRow int; -- Random Row to make a Solar System with
DECLARE CheckSystemExists int; -- Used to insure RandRow is not already a Solar System Row
DECLARE TotalSystemLoops int; -- Total number of loops so each Galaxy gets it's systems.
DECLARE RandomSolarSystemCount int; -- This is the number of Solar Systems that will be in each Galaxy;
DECLARE UpdateSolarCount int;
DECLARE NumberOfOffsets int;

SET CurrentOffsetMultiplier = 0;
SET NumberOfOffsets = 1716;
SET CurrentOffset = 0;


OffsetLoop: LOOP
        SET UpdateSolarCount = 0;
        /*Sets the amount of Solary Systems going in a Galaxy*/
            CheckRandomSolarSystemCount: LOOP
                SET RandomSolarSystemCount = FLOOR(125 + RAND() * (175 - 125) + 1);  
                IF RandomSolarSystemCount >= 125 THEN
                    IF RandomSolarSystemCount <= 175 THEN
                        LEAVE CheckRandomSolarSystemCount;
                    END IF;
                END IF;
            END LOOP;
        UpdateGalaxyWithSolarSystems: LOOP
                SET UpdateSolarCount = UpdateSolarCount + 1;
                IF UpdateSolarCount > RandomSolarSystemCount THEN
                    LEAVE UpdateGalaxyWithSolarSystems;
                END IF;
                        /*Sets RandRow and CheckSystemExists*/
                        CheckExistsLoop: Loop
                            SET RandRow = FLOOR(0 + RAND() * (1000)+ 1);
                            SET CheckSystemExists = (SELECT COUNT(*)
                                FROM
                                    (SELECT * FROM
                                        (SELECT * FROM galaxies2 LIMIT CurrentOffset, 1000) AS LimitedTable
                                    LIMIT RandRow ,1) AS RandTable
                                WHERE SolarSystemName IS NULL);
                            IF CheckSystemExists THEN
                                LEAVE CheckExistsLoop;
                            END IF;
                        END LOOP;

                        /*Updates the tables SolarSystemName column with a default system name*/
                        UPDATE galaxies2 
                            SET SolarSystemName = CONCAT("Solar System ", RandRow)
                            WHERE galaxies2.idGalaxy = 
                                (SELECT LimitedTable.idGalaxy AS GalaxyID FROM
                                    (SELECT galaxies2.idGalaxy FROM galaxies2 LIMIT CurrentOffset, 1000) AS LimitedTable
                                LIMIT RandRow ,1)
                        ;
        END LOOP;
        SET CurrentOffsetMultiplier = CurrentOffsetMultiplier + 1;
        SET CurrentOffset = CurrentOffsetMultiplier * 1000;
        IF CurrentOffsetMultiplier = 1717 THEN
            LEAVE OffsetLoop;
        END IF;
END LOOP;

1 Answer 1

0

It's getting slower and slower because you are "walking" through the galaxies2 table.

SELECT * FROM galaxies2 LIMIT CurrentOffset, 1000

As the CurrentOffset value increases, MySQL has to "walk" through more and more records to get to the starting point. You may actually be able to get a speed boost by specifying an ORDER BY on the primary key. You would want to have an ORDER BY anyway since MySQL just reads records randomly if no order is specified. It does not read the records in any particular order so you could (though unlikely) get the same set of records in different offsets.

It would be better to specify a range on the auto increment field. Assuming you have one. Then the first and last queries should perform about the same. It's not as ideal since there could be gaps from deleted records.

SELECT * FROM galaxies2 WHERE auto_incr_field BETWEEN CurrentOffset AND CurrentOffset+1000
3
  • I get that about slowing down due to reading further in the file, but the idGalaxy is indexed and the queries I have that are being used to watch and keep tabs on this query as it runs show that everything is working as expected. I don't have an auto field other wise I would have just had it randomly choose numbers that way. hmmm... I removed the field in favor of using the id for the sector which to reduce space I used a lettered system. But if I remade with auto increment, then randomized per 1000 and just did the update would that increase the speed since no selects need to be done? Feb 1, 2013 at 16:06
  • An explain on the query should show it is not using an index. Your queries are probably working fine, MySQL just doesn't guarantee a read order unless specified. You can use any index to speed things up: SELECT * FROM g WHERE id LIKE "A%" LIMIT CurrentOffset, 1000 Feb 2, 2013 at 1:58
  • Thanks for sending me in the right direction. I rebuilt the table to include an auto_id. Then I built a temp table for getting the random sectors for each 1000, then using that table and the new table with the auto_id a used a single update statement that updated the new table only where the auto_id matched the random numbers in the table. Worked great. Update finished in like 2 hours. Feb 2, 2013 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.