9

I need to find the word Lämmönmyyntipalvelut from the database. Only, in the database it is in a field, whose value has been a PHP array, converted into JSON using json_encode() and so the special characters are scrabled into hex unicode.

So my query is

SELECT * FROM table WHERE (services LIKE '%Lämmönmyyntipalvelut%')

No results. No surprise. Next, query with special characters converted:

SELECT * FROM table WHERE (services LIKE '%L\u00e4mm\u00f6nmyyntipalvelut%')

No results and I wonder why. Next I tested querying for only special character:

SELECT * FROM table WHERE (services LIKE '%\u00e4%')

Found what was supposed to find. Next I started adding stuff (L to beginning) to see where it went wrong:

SELECT * FROM table WHERE (services LIKE '%L\u00e4%')

No results. Another test:

SELECT * FROM table WHERE (services LIKE '%\u00e4mm%')

Found what was supposed to find.

So my conclusion is that the backslash is somehow messing things up, but I don't understand how?

EDIT:

Exact contents of services field:

["Neuvonta","L\u00e4mm\u00f6nmyyntipalvelut",
"Mets\u00e4-\/energiapuunkorjuupalvelut"]

Exact query:

SELECT id, uid, company_name, services, logo FROM rekisteroeidy_toimijaks 
WHERE 
    (services LIKE '%L\u00e4mm\u00f6nmyyntipalvelut%' AND 
    services LIKE '%Mets\u00e4-\/energiapuunkorjuupalvelut%') 
ORDER BY company_name ASC

I added some line breaks to help readability.

5
  • Show the json_encode()d result. Nov 8, 2012 at 10:40
  • You mean the content of the services field? Nov 8, 2012 at 10:43
  • Yes, maybe someone can see from that, where the problem is. Nov 8, 2012 at 10:45
  • From what I see, Vince's answer should work, if you double every backslash `\` Nov 8, 2012 at 11:03
  • IMO should work too, yes, but still no :/ Nov 8, 2012 at 11:05

3 Answers 3

13

I have absolutely no idea why, but triple escaping helps!

Well, that's only double-escaping, but yes it works and here's why: in MySQL, there is a second layer of escaping involved when you use the LIKE operator.

services LIKE '%L\\\\u00e4mm\\\\u00f6n%'

parsing that MySQL string literal gives you a comparison with the LIKE-query %L\\u00e4mm\\u00f6n%. Because MySQL treats \ in a LIKE query as an escape, that will actually match the literal string containing L\u00e4mm\u00f6n.

The reason for this is so that you can match strings against a query expression that contains a literal % or _ character. For example if I want to search a column for the literal string 100%, I can match it against 100\% (written in a query as '100\\%') and make sure I'm really getting one hundred percent and just not any string starting with a hundred.

It's unfortunate that MySQL uses backslash for both its LIKE query escaping and its string literal escaping, especially given that you're probably writing in an enclosing programming language that also uses them, ending up with actual triple-encoding, which looks like "services LIKE '%L\\\\\\\\u00e4mm\\\\\\\\u00f6n%'" - argh!

It's doubly unfortunate given that this behaviour is not ANSI SQL conformant, and won't work in any other database. ANSI SQL says that there is no escape character in LIKE queries by default, so if you want to match a literal % or _ you have to opt in by nominating an escape character of your own, eg.:

something LIKE '100=%' ESCAPE '='

For cross-database compatibility, it is best always to use the LIKE...ESCAPE form, and pick something other than the horrible backslash! (Aside - MySQL's backslashes for SQL string literal escaping aren't ANSI conformant either! But you can turn that misbehaviour off with the NO_BACKSLASH_ESCAPES sql_mode setting.)

Probably a better idea would be to break services out into a second table rather than squashing them into a single string column - ie. put your schema in First Normal Form. Then you could get a simple lookup of individual values rather than having to do a slow full-table-scan substring-match.

5
  • 1
    For some reason the escaping does not work at all in my database. I tried all combinations with/without BINARY and escaping from 1 time to 8 times. Might have to do something with the utf8mb4_unicode_ci collation. The only way for me to get the example working was doing: SELECT * FROM table WHERE services LIKE BINARY CONCAT('%L', UNHEX('c3a4') ,'mm%'); Also the binary addition is required, or else the result would also match 'Lamm' without diaeresis on the a. Sep 1, 2016 at 8:09
  • @KapiteinWitbaard: that's one way to match an actual ä character (assuming a UTF-8 collation), if you can't just type %Lämm% directly in whatever your environment is. That's not what the OP was doing - they were trying to match an actual backslash in some serialised JSON, not a literal ä. JSON has \uNNNN escapes to reference a Unicode character; SQL literals themselves do not.
    – bobince
    Sep 1, 2016 at 20:49
  • 1
    @KapiteinWitbaard: it should also be more efficient to store the services column in the collation you're going to be using to compare it, if possible. That might be a BINARY collation like utf8_bin if you want to match exact characters; if you want case-insensitive but accent-sensitive matching that's a surprisingly missing combination in the default set of collations, though there are some nasty hacks around it.
    – bobince
    Sep 1, 2016 at 21:09
  • 1
    You are right! I misread the question. I was trying to find posts in my database with a specific unicode character in it. By googling I stumbled on this post, but didn't read thoroughly. Anyways maybe someone can still use my solution if they are not storing data json encoded. In my case I knowingly didn't use a binary column, to be able to match a with ä etc when someone uses the search. But sometimes you want to match ä yourself, and then adding the BINARY to the query solves that issue :-) Sep 4, 2016 at 10:35
  • if you write query in PHP double qoute string , you need to write backslash as "\\\\\\\\" (8 backslash) in LIKE Sep 25, 2021 at 6:09
3

The backslash is a meta character, MySQL understand it this way : "remove the next character and don't parse it as a meta-character".

So you need to escape the backslash :

SELECT * FROM table WHERE (services LIKE '%L\\u00e4%')

Now, MySQL will replace "\\" by "\" (the first backslash is a meta character used to escape the second one)

3
  • Does not help, unfortunately. Nov 8, 2012 at 10:37
  • Really ? In your "Exact query", backslashes are not escaped. Does escaping every backslash like this solves the problem ? : SELECT id, uid, company_name, services, logo FROM rekisteroeidy_toimijaks WHERE (services LIKE '%L\\u00e4mm\\u00f6nmyyntipalvelut%' AND services LIKE '%Mets\\u00e4-\\/energiapuunkorjuupalvelut%') ORDER BY company_name ASC
    – Vince
    Nov 8, 2012 at 10:56
  • Really, as weird as it sounds. Logically your answer is good and correct and makes sense, but I tried and it doesn't work. Nov 8, 2012 at 11:07
1

I have absolutely no idea why, but triple escaping helps!

SELECT id, uid, company_name, services, logo
FROM rekisteroeidy_toimijaks
    WHERE (
    services LIKE  '%L\\\\u00e4mm\\\\u00f6n%'
)
ORDER BY company_name ASC 
LIMIT 0 , 30
4
  • 1
    This sounds like you're doing it with php, java or some other language. I'm pretty sure, Vince's answer was done without that. So, if this is the case, you should accept his answer. Nov 8, 2012 at 14:30
  • But how can that be dependent on the language I'm using (it's PHP, correct), when I get exactly the same behavior using phpMyAdmin? Nov 8, 2012 at 16:12
  • I just tried this myself in command line mysql and you're right. I stand corrected, sorry for the confusion. Nov 8, 2012 at 16:23
  • Because when you put the SQL statement string into a PHP string, the string is escaped multiple time : first time by PHP (string declaration), and second time by MySQL. As far as PHP and many other languages (Java, C#...) use the backslash as the escaping character, you'll encounter this trouble and need double escape.
    – Vince
    Nov 8, 2012 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.