4

I can't figure out what I'm doing wrong. I'm getting file content from the database. When I echo the content, everything displays just fine, when I write it to a file (.html) it breaks. I've tried iconv and a few other solutions, but I just don't understand what I should put for the first parameter, I've tried blanks, and that didn't work very well either. I assume it's coming out of the DB as UTF-8 if it's echoing properly. Been stuck a little while now without much luck.

function file($fileName, $content) {
    if (!file_exists("out/".$fileName)) {
        $file_handle = fopen(DOCROOT . "out/".$fileName, "wb") or die("can't open file");
        fwrite($file_handle, iconv('UTF-8', 'UTF-8', $content));
        fclose($file_handle);
        return TRUE;
    } else {
        return FALSE;
    }
}

Source of the html file looks like.

Comes out of the DB like this:

<h5>Текущая стабильная версия CMS</h5>

goes in file like this

<h5>Ð¢ÐµÐºÑƒÑ‰Ð°Ñ ÑÑ‚Ð°Ð±Ð¸Ð»ÑŒÐ½Ð°Ñ Ð²ÐµÑ€ÑÐ¸Ñ CMS</h5>

EDIT:

Turns out the root of the problem was Apache serving the files incorrectly. Adding

AddDefaultCharset utf-8

To my .htaccess file fixed it. Hours wasted... At least I learned something though.

19
  • Put your output HTML somewhere. Chances are you have not added the UTF-8 <meta> tag header to the HTML.
    – Dean
    Nov 19, 2015 at 22:01
  • @Dean would that mater even when looking at the source? <li>Ðа форуме чаÑто упоминаетÑÑ ÐºÐ¾Ð´Ð¾Ð³ÐµÐ½ÐµÑ€Ð°Ñ‚Ð¾Ñ€. </li>
    – Serhiy
    Nov 19, 2015 at 22:04
  • 1
    I bet that Ð¢ÐµÐºÑƒÑ‰Ð°Ñ ÑÑ‚Ð°Ð±Ð¸Ð»ÑŒÐ½Ð°Ñ Ð²ÐµÑ€ÑÐ¸Ñ is the UTF-8 of Текущая стабильная версия seen as ISO-8859-1, also because the character Ð is 0xD0 in ISO-8859-1, and 0xD0 is the first of the 2 UTF-8 bytes of the letters of the cyrillic alphabet. This means that you ARE writing UTF-8, but are looking at it as if it were ISO-8859-1 (or ISO-8859-15). Nov 19, 2015 at 23:38
  • 1
    @Martin The Walter Tross comment you pointed me towards. He said it's the reader. So after I outputed the file via cat, I got good results, and figured my reader, aka Apache, was at fault. Next thing I did was google Apache and UTF-8 and got stackoverflow.com/questions/913869/…
    – Serhiy
    Nov 20, 2015 at 0:16
  • 1
    Glad I could help. I have to correct my comment though: Characters in the cyrillic block of Unicode start with one of 0xD0, 0xD1, 0xD2, 0xD3 when encoded as UTF-8 (because they are in the range U+0400 to U+04FF). These 4 start bytes appear as Ð, Ñ, Ò and Ó in ISO-8859-1 (with decreasing frequency). Nov 20, 2015 at 7:16

2 Answers 2

1

Edit: The database encoding does not seem to be the issue here, so this part of the answer is retained for information only

I assume it's coming out of the DB as UTF-8

This is most likely your problem, what database type do you use? Have you set the character encoding and collation details for the database, the table, the connection and the transfer.

If I was to hazard a guess, I would say your table is MySQL and that your MySQL collation for the database / table / column should all be UTF8_general_ci ?

However, for some reason MySQL UTF8 is not actually UTF8, as it stores its data in 3bits rather than 4bits, so can not store the whole UTF-8 Character sets, see UTF-8 all the way through .

So you need to go through every table, column on your MySQL and change it from UTF8_ to the UTF8mb4_ (note: since MySQL 5.5.3) which is UTF8_multibyte_4 which covers the whole UTF-8 Spectrum of characters.

Also if you do any PHP work on the data strings be aware you should be using mb_ PHP functions for multibyte encodings.

And finally, you need to specify a connection character set for the database, don't run with the default one as it will almost certainly not be UTF8mb4, and hence you can have the correct data in the database, but then that data is repackaged as 3bit UTF8 before then being treated as 4bit UTF8 by PHP at the other end.

Hope this helps, and if your DB is not MySQL, let us know what it is!

Edit:

function file($fileName, $content) {
    if (!file_exists("out/".$fileName)) {
        $file_handle = fopen(DOCROOT . "out/".$fileName, "wb") or die("can't open file");
        fwrite($file_handle, iconv('UTF-8', 'UTF-8', $content));
        fclose($file_handle);
        return TRUE;
    } else {
        return FALSE;
    }
}
  • your $file_handle is trying to open a file inside an if statement that will only run if the file does not exist.

  • Your iconv is worthless here, turning from "utf-8" to er, "utf-8". character detection is extremely haphazard and hard for programs to do correctly so it's generally advised not to try and work out / guess what a character encoding it, you need to know what it is and tell the function what it is.

14
  • Arg, still failing, switched the table / connection to be UTF8mb4, re-inserted the data just to make sure, commented out the str_replace just in case, took out the iconv, made "wb" just "w" and am still failing.
    – Serhiy
    Nov 19, 2015 at 22:59
  • you will need to check that the connection between PHP and MySQL is also utf8mb4, something like $dbconnection->set_charset("utf8mb4");
    – Martin
    Nov 19, 2015 at 23:02
  • if that also doesn't help you, then we'd need to see what the code looks like on your screen output (when it's ok) and compare that to what it looks like on your file (which you put in your question already). Also if you have changed the database to use UTF8mb4 then you will ideally need to re-add the code to the database so it can be saved in 4bit rather than 3bit UTF8. @Serhiy
    – Martin
    Nov 19, 2015 at 23:04
  • Yep, re-inserted the data, had the set_charset thing, when I echo, everything is perfect, when I write to file, it's not. I honestly feel like it must be the file writing part. I am initially getting the data from CURL, but since it's getting in the DB and out just fine, there's no way that could be the culprit could it? pastebin.com/raw.php?i=EshBmV77
    – Serhiy
    Nov 19, 2015 at 23:07
  • what is the character encoding if your output HTML file when it works? I take it that is UTF-8 (I'm just ticking off causes it isn't :-D )
    – Martin
    Nov 19, 2015 at 23:08
0

The comment by Dean is actually very important. The HTML should have a <meta charset="UTF-8"> inside <head>.

That iconv call is actually not useful and, if you are right that you are getting your content as UTF-8, it is not necessary.

You should check the character set of your database connection. Your database can be encoded in UTF-8 but the connection could be in another character set.

Good luck!

4
  • Browsers should detect UTF-8 without the meta, it's not a necessary requirement. I think the issue is with the database encoding.
    – Martin
    Nov 19, 2015 at 22:34
  • Yeah, I felt like Dean's comment was a tangent. The file is bad, when I open it in a text editor, no browser needed.
    – Serhiy
    Nov 19, 2015 at 22:43
  • 1
    @Serhiy I think it's your binary marker on your fopen that is the issue. See my answer (at the bottom)
    – Martin
    Nov 19, 2015 at 22:44
  • @Martin, thanks, I am reading through it, it's a bit much to take in at once, going through it, thanks for such details.
    – Serhiy
    Nov 19, 2015 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.