I'm working with a MySQL database that has some data imported from Excel. The data contains non-ASCII characters (em dashes, etc.) as well as hidden carriage returns or line feeds. Is there a way to find these records using MySQL?
-
8Ollie Jones has a much better answer (check the bottom).– Jonathan ArkellCommented Nov 12, 2012 at 17:45
-
1@JonathanArkell Not on the bottom anymore :)– BrilliandCommented May 22, 2014 at 22:02
-
Correction.. check the middle! ;)– Jonathan ArkellCommented May 23, 2014 at 15:26
-
This is the answer @Jonathan is talking about stackoverflow.com/a/11741314/792066– BraiamCommented Oct 8, 2018 at 19:53
10 Answers
MySQL provides comprehensive character set management that can help with this kind of problem.
SELECT whatever
FROM tableName
WHERE columnToCheck <> CONVERT(columnToCheck USING ASCII)
The CONVERT(col USING charset)
function turns the unconvertable characters into replacement characters. Then, the converted and unconverted text will be unequal.
See this for more discussion. https://dev.mysql.com/doc/refman/8.0/en/charset-repertoire.html
You can use any character set name you wish in place of ASCII. For example, if you want to find out which characters won't render correctly in code page 1257 (Lithuanian, Latvian, Estonian) use CONVERT(columnToCheck USING cp1257)
-
24This is an excellent solution to this problem and much more robust. Commented Oct 23, 2012 at 14:33
-
6this is also useful to find characters with accents (á ä etc) or character not belonging to encoding Commented Oct 22, 2014 at 12:34
-
3much better than using REGEXP (which doesn't seem to work for me for finding accents) and also provides a simple mechanism for making everything ascii again...– D2TheCCommented Nov 14, 2017 at 8:47
-
1This answer works wonderfully and will bring up strings that contain any non-ASCII characters rather than just strings that contain only non-ASCII characters. Thank you!– techjpCommented Sep 11, 2018 at 7:26
-
3
You can define ASCII as all characters that have a decimal value of 0 - 127 (0x00 - 0x7F) and find columns with non-ASCII characters using the following query
SELECT * FROM TABLE WHERE NOT HEX(COLUMN) REGEXP '^([0-7][0-9A-F])*$';
This was the most comprehensive query I could come up with.
-
4Best answer so far, but it's even easier like this :
SELECT * FROM table WHERE LENGTH( column ) != CHAR_LENGTH( column )
– SuNCommented Nov 28, 2012 at 18:53 -
16-1 This can yield erroneous results. Suppose, for example, that one has a UTF-16 column containing
'ā'
(encoded by the byte sequence0x0101
) - it would be deemed "ASCII" using this test: a false negative; indeed, some character sets do not encode ASCII characters within0x00
to0x7f
whereupon this solution would yield a false positive. DO NOT RELY UPON THIS ANSWER!– eggyalCommented Feb 11, 2014 at 9:13 -
2@sun: That doesn't help at all - many character sets are fixed-length and so
LENGTH(column)
will be a constant multiple ofCHAR_LENGTH(column)
irrespective of the value.– eggyalCommented Feb 11, 2014 at 9:39
It depends exactly what you're defining as "ASCII", but I would suggest trying a variant of a query like this:
SELECT * FROM tableName WHERE columnToCheck NOT REGEXP '[A-Za-z0-9]';
That query will return all rows where columnToCheck contains any non-alphanumeric characters. If you have other characters that are acceptable, add them to the character class in the regular expression. For example, if periods, commas, and hyphens are OK, change the query to:
SELECT * FROM tableName WHERE columnToCheck NOT REGEXP '[A-Za-z0-9.,-]';
The most relevant page of the MySQL documentation is probably 12.5.2 Regular Expressions.
-
3Shouldn't you escape the hyphen and period? (Since they do have special meanings in a regular expression.) SELECT * FROM tableName WHERE NOT columnToCheck REGEXP '[A-Za-z0-9\.,\-]';– TooonyCommented Mar 12, 2009 at 16:03
-
3@Tooony No, inside of a set, a period just means itself and the dash only has special meaning between other characters. At the end of the set, it means only itself. Commented Jan 25, 2012 at 21:55
-
12This query only finds all lines in tableName that do not contain an alphanumeric character. This does not answer the question. Commented Aug 26, 2013 at 17:26
-
12That is for columns that don't have any ascii characters at all, so it will miss those with a mix of ascii and non-ascii characters. The answer below from zende checks for one or more non-ascii characters. This helped me for the most part
SELECT * FROM tbl WHERE colname NOT REGEXP '^[A-Za-z0-9\.,@&\(\) \-]*$';
Commented Oct 16, 2015 at 21:00 -
2This only works (for me anyway) to find strings that contain NONE of those characters. It does not find strings that contain a mix of ASCII and non-ASCII characters.– techjpCommented Sep 11, 2018 at 7:24
This is probably what you're looking for:
select * from TABLE where COLUMN regexp '[^ -~]';
It should return all rows where COLUMN contains non-ASCII characters (or non-printable ASCII characters such as newline).
-
7Works great for me. "regexp '[^ -~]'" means has a character that is before space " " or after "~" or ASCII 32 - 126. All letters, numbers, and symbols, but no unprintable things.– JoshCommented Dec 11, 2012 at 2:18
-
You can even get it as a tee-shirt ;) catonmat.net/blog/my-favorite-regex– SamGoodyCommented Feb 7, 2013 at 20:52
-
2Note the warning in the documentation: "The
REGEXP
andRLIKE
operators work in byte-wise fashion, so they are not multi-byte safe and may produce unexpected results with multi-byte character sets. In addition, these operators compare characters by their byte values and accented characters may not compare as equal even if a given collation treats them as equal."– eggyalCommented Feb 11, 2014 at 9:36 -
1thanks for this. what i'm wondering is how to replace a replacement character - e.g. â��– mars-oCommented Apr 15, 2014 at 14:58
-
1@mars-o - the black diamond indicates an invalid utf8 character. More discussion here Commented Apr 8, 2017 at 19:57
One missing character from everyone's examples above is the termination character (\0). This is invisible to the MySQL console output and is not discoverable by any of the queries heretofore mentioned. The query to find it is simply:
select * from TABLE where COLUMN like '%\0%';
Based on the correct answer, but taking into account ASCII control characters as well, the solution that worked for me is this:
SELECT * FROM `table` WHERE NOT `field` REGEXP "[\\x00-\\xFF]|^$";
It does the same thing: searches for violations of the ASCII range in a column, but lets you search for control characters too, since it uses hexadecimal notation for code points. Since there is no comparison or conversion (unlike @Ollie's answer), this should be significantly faster, too. (Especially if MySQL does early-termination on the regex query, which it definitely should.)
It also avoids returning fields that are zero-length. If you want a slightly-longer version that might perform better, you can use this instead:
SELECT * FROM `table` WHERE `field` <> "" AND NOT `field` REGEXP "[\\x00-\\xFF]";
It does a separate check for length to avoid zero-length results, without considering them for a regex pass. Depending on the number of zero-length entries you have, this could be significantly faster.
Note that if your default character set is something bizarre where 0x00-0xFF don't map to the same values as ASCII (is there such a character set in existence anywhere?), this would return a false positive. Otherwise, enjoy!
-
100-FF includes all possible 8-bit values, which is what
REGEXP
is checking. Hence it is guaranteed to always match. Also^$
is probably not what you wanted. Commented Apr 8, 2017 at 19:55 -
Definitely the best REGEXP solution for finding all 8 bit characters but not as good as the CONVERT(col USING charset) solution which also will allows control characters while limiting display characters to a specific charset.– techjpCommented Dec 25, 2018 at 20:07
Try Using this query for searching special character records
SELECT *
FROM tableName
WHERE fieldName REGEXP '[^a-zA-Z0-9@:. \'\-`,\&]'
@zende's answer was the only one that covered columns with a mix of ascii and non ascii characters, but it also had that problematic hex thing. I used this:
SELECT * FROM `table` WHERE NOT `column` REGEXP '^[ -~]+$' AND `column` !=''
In Oracle we can use below.
SELECT * FROM TABLE_A WHERE ASCIISTR(COLUMN_A) <> COLUMN_A;
for this question we can also use this method :
Question from sql zoo:
Find all details of the prize won by PETER GRÜNBERG
Non-ASCII characters
ans: select*from nobel where winner like'P% GR%_%berg';
-
2