72

I'm using mysqldump in a cron job to backup a database with over 2 million rows.

It creates a text file which can be used to restore the datalog from the command line.

I thought it would be useful to edit the dump before a restore as a quick way of changing values and table or column names - at least until I learn more and become confident about doing it with ALTER and UPDATE.

Editing large text files does not bother me, but I was surprised to find that in a 250 megabyte dump of my database, there were only about 300 lines. Each line was something like 800k characters long.

Is there another way of generating dumps with more control over line length?

Or should I post-process the dump with tools like sed or Perl?

6
  • Consider the ; between queries as your line endian.
    – Havenard
    Aug 18, 2009 at 12:40
  • Actually, I used Perl and said $line =~ s{\).\(}{),\n(}g; so I got LOTS of extra lines.
    – pavium
    Aug 18, 2009 at 13:29
  • That is to say, the ; was already my eol at the end of each INSERT into table_name VALUES (..),(..),(..) .. (..); 282 lines in the entire file, 252 ending with ; I chose to insert newlines after the commas
    – pavium
    Aug 18, 2009 at 13:53
  • 2
    You're sure your actual data had no parenthesis-comma-parenthesis sequences in it? Dec 22, 2012 at 0:22
  • 4
    This is a known shortcoming in mysqldump. It was first reported in 2004. In 2011, both Tim Riker & Lon Binder suggested a 1-line patch to fix it. Mindbogglingly, this still has not been implemented by the mysqldump developers/maintainers. Because the original bug report was closed (wrongly and unhelpfully), the issue is now being tracked here.
    – user82216
    Jun 3, 2018 at 15:59

8 Answers 8

82

By default, mysqldump generates only one INSERT command per table, resulting in one (very long) line of inserted data for each table that got dumped. This is essentially because the "batch" inserts are much faster than if it generated a separate INSERT query for every record in every table.

So, it's not that mysqldump has created arbitrarily long lines, and you can just impose some other cutoff length. The lines are long for a reason.

If it's really important to get the INSERTs broken down onto multiple lines, you can indicate that with:

mysqldump --extended-insert=FALSE --complete-insert=TRUE ...

Note, however, that restoring tables will take longer in this format.

12
  • 2
    I'm wondering if anyone can confirm that the net_buffer_length option will also work well. It seems like a better solution to me. Dec 1, 2012 at 0:22
  • 36
    @VoteyDisciple - that explains why it's one big INSERT, but not why it's one big line. There could easily be a newline after each insertion row, right? Dec 22, 2012 at 0:26
  • 9
    Oh, there sure IS a demand for that! We use mysqldump to store DB for each "tag". We make some smaller updates between tags and with one big line you can't see which content has changed and you need to scroll sideways to it. Mar 22, 2013 at 13:08
  • 18
    In my eyes it is just a "formatting issue". Why not generate ONE INSERT statement but the value blocks for each row in separate lines. This would make the generated file human readable and does not effect performance at all.
    – hfrmobile
    Jul 8, 2013 at 8:04
  • 2
    PHPMyAdmin generates its dump files with a new line between each insertion. I guess there was demand for that.
    – Tamlyn
    Mar 27, 2014 at 10:37
35

I was browsing the MySQL source code looking for a solution to this problem today. The maximum line length is enforced by the variable opt_net_buffer_length which is supposed to match the MySQL server's buffer size. It is comically large.

But anyway, it's an option, so just do this:

mysqldump --net_buffer_length=5000 ...

The minimum value is 4096.

6
  • 8
    Will this cause problems with individual rows containing more than 4K data? Oct 8, 2012 at 19:50
  • @CyberShadow good question. I bet it would but I have no idea. Next time I do this, I'll try matching up the mysqldump net_buffer_length to the max_allowed_packet and see if that works. I just did an import where 16M max_allowed_packet was exceeded by query size. Even 64M was too small - went to 256M and got it through. So I would probably try a net_buffer_length of around 10M next time, myself, and see what that does. Dec 1, 2012 at 0:20
  • 1
    @CyberShadow no it doesn't cause problems with bigger rows - they take as much space as they need. Tested with MariaDB 10.0.21. Nov 27, 2015 at 15:55
  • 1
    Performance wise, it took to import my 20MB / 335k row database dump: [ with a single-transaction dump: 22s ] // [ with a 4096 buffer length dump: 1m 44s ] // [ with extended-insert=false: ???? I got bored waiting and CTRL-Ced it after 33 minutes ]. So this answer is without a doubt the best compromise between speed and diffability. Caveat emptor: My subsequent dump was identical but I didn't test with row data longer than 4096 chars.
    – Idris
    Jul 16, 2017 at 14:08
  • 1
    *Actually, thinking about it, if a row early on in a table is modified or deleted, then all subsequent lines would have their content offset across the block boundaries, making subsequent dumps differ on every line after that point. So maybe this doesn't have great diffability after all...
    – Idris
    Jul 16, 2017 at 14:28
22

I came across an answer on the MySQL forums, which conclusively shows adding '\n' after each INSERT group is not possible using mysqldump alone, without modifying the source:

The extended format cannot be 100% properly parsed based on the comma or parenthesis, you would to count fields. The best solution, fix mysqldump to linebreak on output.

Very minor change: On line 3506, you can see where the row-ending comma is output:
fputc(',',md_result_file); /* Always row break */

Simply insert this line immediately after line 3506:
fputc('\n',md_result_file); /* Lon Binder says wrap that line! */

re-compile and done.

@see http://forums.mysql.com/read.php?28,420002,426110#msg-426110

Thanks Lon B!

(I've included the content from the MySQL forum just in case the forum disappears.)

5
  • 17
    Ugh, why is this not an option on mysqldump? This is the best of both worlds.
    – SineSwiper
    Nov 22, 2013 at 15:23
  • Please give link to source of mysqldump. Aug 27, 2014 at 6:42
  • @Speakus - MySQL is owned by Oracle and offers no direct link to download or view individual source files - you can download the source bundle from here: dev.mysql.com/downloads/utilities - just select 'Source' as the platform.
    – StampyCode
    Aug 27, 2014 at 7:40
  • thank you. minor improve for your comment: i should choose source not from your link, but from here: dev.mysql.com/downloads/mysql + open file mysql-5.6.20/client/mysqldump.c + search required line via comment (i have found line 3791) + add new line as described. another way to get source code: mariadb.com/kb/en/mariadb/documentation/getting-started/… + change maria/client/mysqldump.c (line 3928). Unfortunately, both tools (oracle mysql and mariadb) don't have option to solve question out of box still Aug 27, 2014 at 12:46
  • 2022 now: Has such a "best of both worlds" option been added to the mysqldump tool meanwhile?
    – porg
    Dec 20, 2022 at 0:16
6

This flag is also worked:

mysqldump --skip-extended-insert 

Just same as --extended-insert=FALSE.

2
  • 1
    Yeah I like that one better for some reason :) Just needed this for a diff, where the normal long lines make diffs really painful. Jul 13, 2017 at 12:38
  • Every record starts with INSERT INTO table_name VALUES (...);, which takes up more space, but can lend some helpful context if it's not obvious from the data which table it's going into.
    – mwfearnley
    Apr 29, 2019 at 8:46
5

Using a regex to split lines is not enough, you need a parser that will properly understand quotes and escaping characters.

I just wrote a parser since I couldn’t find one: http://blog.lavoie.sl/2014/06/split-mysqldump-extended-inserts.html

1
  • Very useful for existing dumps :) Aug 4, 2020 at 7:34
1

If you have already a SQL dump with very long lines and want to read it easily, you can use

cat dump.sql  | sed 's$VALUES ($VALUES\n($g' | sed 's$),($),\n($g' | less

cat dump.sql  | sed 's$VALUES ($VALUES\n($g' | sed 's$),($),\n($g' | grep something
1

This is how I solved this problem:

  1. I ask mysqldump --skip-extended-insert to produce one INSERT INTO per line:
INSERT INTO `action_events` VALUES (1,'8cecd83d-76b7-44d3-ad6a-5ad3ff6410dd');
INSERT INTO `action_events` VALUES (2,'8cecd8ba-465c-4d2e-b9f0-caf504817e3a');
INSERT INTO `action_events` VALUES (3,'8cecd961-03d1-461f-8425-067c1d01d238');
INSERT INTO `action_events` VALUES (4,'8cf08433-f262-443c-90d1-629dcd11c5aa');
INSERT INTO `xx` VALUES (5,'8cf08433-f262-443c-90d1-629dcd11c5aa');
INSERT INTO `yy` VALUES (6,'8cf08433-f262-443c-90d1-629dcd11c5aa');
  1. Using awk I join consecutive lines with INSERT INTO by commma:
INSERT INTO `action_events` VALUES
(1,'8cecd83d-76b7-44d3-ad6a-5ad3ff6410dd'),
(2,'8cecd8ba-465c-4d2e-b9f0-caf504817e3a'),
(3,'8cecd961-03d1-461f-8425-067c1d01d238'),
(4,'8cf08433-f262-443c-90d1-629dcd11c5aa');
INSERT INTO `xx` VALUES
(5,'8cf08433-f262-443c-90d1-629dcd11c5aa');
INSERT INTO `yy` VALUES
(6,'8cf08433-f262-443c-90d1-629dcd11c5aa');

Here is the script join-insert-into:

#!/bin/bash

# 1. print $0
# 2. print INSERT INTO
# 3. print (1,2,3),
# 4. print (1,2,3);
awk '
    BEGIN { FS = "\n"; OFS = ""; max_allowed_packet = 25*1024*1024; }
    END { if (prev_values) print prev_values ";" }
    {
        if (match($0, /^INSERT INTO [^ ]+ VALUES /)) {
            table = substr($0, RSTART, RLENGTH - 1)
            values = substr($0, RLENGTH + 1, length($0) - RLENGTH - 1)
            if (prev_table != table) {
                if (prev_values) {
                    print prev_values ";"
                }
                print table
                packet = length(table) + 1
            }
            else {
                packet += length(prev_values) + 2
                if (packet >= max_allowed_packet) {
                    print prev_values ";"
                    print table
                    packet = length(table) + 1
                }
                else {
                    print prev_values ","
                }
            }
            prev_table = table
            prev_values = values
        }
        else {
            if (prev_values) {
                print prev_values ";"
                prev_table = null
                prev_values = null
            }
            print $0
        }
    }
'

It can be used like that:

# https://stackoverflow.com/a/34670902/1478566
MYSQL_PWD="$DB_PASSWORD" mysqldump \
    --no-tablespaces --quick --skip-extended-insert \
    -h$DB_HOST --port=${DB_PORT:-3306} \
    -u$DB_USERNAME $DB_DATABASE "$@" | join-insert-into
-4

Post-process the dump file with python. You may be happier than perl or sed.

If you're running on Linux, you already have it installed. If you're running on Windows, the installer is painless.

Before that, however, learn to use SQL UPDATE and SQL ALTER. You'll be happiest doing things the right way.

6
  • I prefer Perl or sed, simply because that's what I'm used to. And although there's a certain satisfaction in editing a big file, I will learn the SQL commands, I promise.
    – pavium
    Aug 18, 2009 at 13:29
  • 9
    Post-processing is a bad idea unless you're really certain you understand exactly how the data is escaped, and exactly how to write code to deal with it. Otherwise you're going to corrupt your data. Dec 22, 2012 at 0:20
  • 3
    If you're sure you won't corrupt the data, here's an example command to split the inserts mysqldump -udbuser dbname | sed 's/),(/),\n(/g' > dump.sql
    – Tamlyn
    Mar 27, 2014 at 10:32
  • 2
    @Tamlyn But what if I have ),( in a VARCHAR field? Maybe a comma delimited list of emoticons, or something. I guess, yeah... if you know your data well enough that there's no possibility that you have anything like that...
    – James S
    May 30, 2014 at 20:38
  • It might be better to post a Python solution or a link to one, rather than just the Python homepage. Someone who hasn't heard of Python before will have a long journey ahead before they can write a suitable post-processing script.
    – mwfearnley
    May 3, 2019 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.