6

I'm trying to save some values in MySQL database by using Hibernate, but most Lithuanian characters won't get saved, including ąĄ čČ ęĘ ėĖ įĮ ųŲ ūŪ(they are saved as ?), however, šŠ žŽ do get saved.

If I do inserts manually, then those values are properly saved, so the problem is most likely in Hibernate configuration.

What I have tried so far:

hibernate.charset=UTF-8
hibernate.character_encoding=UTF-8
hibernate.use_unicode=true

---------

properties.put(PROPERTY_NAME_HIBERNATE_USE_UNICODE,
            env.getRequiredProperty(PROPERTY_NAME_HIBERNATE_USE_UNICODE));
    properties.put(PROPERTY_NAME_HIBERNATE_CHARSET,
            env.getRequiredProperty(PROPERTY_NAME_HIBERNATE_CHARSET));
    properties
            .put(PROPERTY_NAME_HIBERNATE_CHARACTER_ENCODING,
                    env.getRequiredProperty(PROPERTY_NAME_HIBERNATE_CHARACTER_ENCODING));

---------

private void registerCharachterEncodingFilter(ServletContext aContext) {
    CharacterEncodingFilter cef = new CharacterEncodingFilter();
    cef.setForceEncoding(true);
    cef.setEncoding("UTF-8");
    aContext.addFilter("charachterEncodingFilter", cef)
            .addMappingForUrlPatterns(null, true, "/*");
}

As described here

I tried adding ?useUnicode=true&characterEncoding=utf-8 to db connection url.

As described here

I ensured that my db is set to UTF-8 charset. phpmyadmin > information_schema > schemata

def db_name utf8 utf8_lithuanian_ci NULL

This is how I save into db:

//Controller
buildingService.addBuildings(schema.getBuildings());
        List<Building> buildings = buildingService.getBuildings();
        System.out.println("-----------");
        for (Building b : schema.getBuildings()) {
            System.out.println(b.toString());
        }
        System.out.println("-----------");
        for (Building b : buildings) {
            System.out.println(b.toString());
        }
        System.out.println("-----------");

//Service:
@Override
public void addBuildings(List<Building> buildings) {
    for (Building b : buildings) {
        getCurrentSession().saveOrUpdate(b);
    }
}

First set of println contains all Lithuanian characters, while second replaces most with ?

EDIT: Added details

insert into buildings values (11,'ąĄčČęĘ', 'asda');    
select short, hex(short) from buildings;
//Šalt. was inserted via hibernate
//letters are properly displayed:
ąĄčČęĘ       | C485C484C48DC48CC499C498
MIF Šalt.    | 4D494620C5A0616C742E  

select address, hex(address) from buildings;
 Šaltini? <...> | C5A0616C74696E693F20672E2031412C2056696C6E697573
//should contain "ų"
--------
show create table buildings;
buildings | CREATE TABLE `buildings` (
  `id` int(11) NOT NULL,
  `short` varchar(255) COLLATE utf8_lithuanian_ci DEFAULT NULL,
  `address` varchar(255) COLLATE utf8_lithuanian_ci DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_lithuanian_ci 

EDIT: I did not find a proper solution, so I came up with a workaround. I ended up escaping/unescaping characters, storing them like this: \uXXXX.

1
  • This doesn't seem to be your specific issue, but we had a similar problem where an Ant build script was running a sql task and was defaulting to the encoding scheme of the host machine, which was cp1252 on windows server 2008. Even though the rest of our configuration all throughout the process specified UTF-8, this build script defaulted to cp1252 when run on the production machine (but not in dev environments). Super mysterious until we figured out what was going on. Apr 7, 2015 at 16:27

3 Answers 3

3

Let's verify that they were stored correctly... Please do SELECT col, HEX(col) ... to fetch some cell with Lithuanian characters. A correctly stored ą will show C485. The others should show various hex values of C4xx or C5xx. 3F is ?.

But, more importantly, 4 characters do show. Š should be C5A0 if properly stored as utf8. However, I suspect, you will see 8A, implying that the column in the table is really declared as CHARACTER SET latin1. (The 4 characters show up in the first column of my charset blog ).

Do SHOW CREATE TABLE to see how the column is defined. If it says latin1, then the problem is with the table definition, and you probably ought to start over.

3
  • Latin Small Letter U with Ogonek (ų) should have hex C5B3, so it should have gone the same as the other letters with ogoneks.
    – Rick James
    Mar 30, 2015 at 0:57
  • 2
    The hex that did so, looks like correct Lithuanian characters. I suspect that the ų was inserted some time ago, when you had something configured differently. (Notice that the Š came through.) Was insert into buildings values (11,'ąĄčČęĘ', 'asda'); done through Hibernate? It seems to have stored correctly.
    – Rick James
    Mar 30, 2015 at 1:01
  • No, I inserted that manually. In current dev environment, database gets recreated every time.
    – Marius
    Mar 30, 2015 at 6:19
0

You have to ensure that every component taking part in data entry uses UTF-8 encoding explicitly.

  • If you enter the values via browser, make sure that the page displaying the results with the following header Content-Type: text/html; charset=utf-8.

  • The input form is defined as follows

    <form action="submit" accept-charset="UTF-8">...</form>.

  • If you are creating String objects from byte array, make sure you explicitly state the Charset in the constructor.

  • If your entry happens from a text file, that file has to be UTF-8 encoded.

  • If it is hardcoded directly in your code, then the source has to be UTF-8 encoded.

0

The fact that your DB holds correct UTF-8 (two or more bytes for a special letter) is reassuring.

If you get one single ? for a special letter, it was attempted to do a UTF-8 conversion to some encoding that does not contain those letters. And that seems to be the case. The letters that are converted correctly are in the ISO-8859-1 or Windows-1252 range. The others are not. Now ISO-88591-1 aka Latin-1 is the default HTTP encoding, default in java EE server. You might like to do before writing:

response.setCharacterEncoding("UTF-8");

Now one problem with System.out.println is that it uses the system default encoding. Logging to a file with a logger is more interesting. Or debugging and inspecting the String and its char array.

That the schema does seemingly work, may be that the schema Strings stem immediately from a Java source, and the editor encoding and javac compiler encoding differ. This can be checked by u-escaping the string literals in java: "\u0105" instead of "ą".

Make a unit test that writes and reads from the database.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.