58

I have php document signup.php which save the content from form (in form.php document) to MySQL base. The problem arises when I want to reformat the input content. I want do decode UTF-8 charachters like à->a.

  $first_name=$_POST['first_name'];
  $last_name=$_POST['last_name'];
  $course=$_POST['course'];

  $chain="prêt-à-porter";

$pattern = array("'é'", "'è'", "'ë'", "'ê'", "'É'", "'È'", "'Ë'", "'Ê'", "'á'", "'à'", "'ä'", "'â'", "'å'", "'Á'", "'À'", "'Ä'", "'Â'", "'Å'", "'ó'", "'ò'", "'ö'", "'ô'", "'Ó'", "'Ò'", "'Ö'", "'Ô'", "'í'", "'ì'", "'ï'", "'î'", "'Í'", "'Ì'", "'Ï'", "'Î'", "'ú'", "'ù'", "'ü'", "'û'", "'Ú'", "'Ù'", "'Ü'", "'Û'", "'ý'", "'ÿ'", "'Ý'", "'ø'", "'Ø'", "'œ'", "'Œ'", "'Æ'", "'ç'", "'Ç'");

$replace = array('e', 'e', 'e', 'e', 'E', 'E', 'E', 'E', 'a', 'a', 'a', 'a', 'a', 'A', 'A', 'A', 'A', 'A', 'o', 'o', 'o', 'o', 'O', 'O', 'O', 'O', 'i', 'i', 'i', 'I', 'I', 'I', 'I', 'I', 'u', 'u', 'u', 'u', 'U', 'U', 'U', 'U', 'y', 'y', 'Y', 'o', 'O', 'a', 'A', 'A', 'c', 'C'); 

$chain = preg_replace($pattern, $replace, $chain);

echo $chain; // print pret-a-porter

$first_name =  preg_replace($pattern, $replace, $first_name);

echo $first_name; // does not change the input!?!

Why it works perfectly for $chain, but for $first_name or $last_name doesnt work?

Also i try

echo $first_name; // print áááááábéééééébšššš
$trans = array("á" => "a", "é" => "e", "š" => "s");
echo strtr("áááááábéééééébšššš", $trans); // print aaaaaabeeeeeebssss
echo strtr($first_name,$trans);  // print áááááábéééééébšššš

but the problem, as you can see, is same!

4
  • 2
    Does your file have the correct encoding? Does the stuff in $_POST have the same encoding? (i.e. the one you set for the browser to display the page with?) You may also want to set the unicode flag for preg_replace
    – phant0m
    Apr 14, 2012 at 10:36
  • 1
    As phant0m mentions, you need to keep close watch on all your encodings. Default encoding with POST is ISO-8859-1 unless you tell it specifically which you want. (accept-charset='UTF-8')
    – danp
    Apr 14, 2012 at 10:41
  • added a couple of example functions :)
    – danp
    Apr 14, 2012 at 10:58
  • 1
    Thank you very much for your help, when I add accept-charset="UTF-8" at the <form action="sign_up.php" method="post" accept-charset="UTF-8"> code working perfectly. Apr 14, 2012 at 11:07

9 Answers 9

99

There's a much easier way to do this, using iconv - from the user notes, this seems to be what you want to do: characters transliteration

// PHP.net User notes
<?php
    $string = "ʿABBĀSĀBĀD";

    echo iconv('UTF-8', 'ISO-8859-1//TRANSLIT', $string);
    // output: [nothing, and you get a notice]

    echo iconv('UTF-8', 'ISO-8859-1//IGNORE', $string);
    // output: ABBSBD

    echo iconv('UTF-8', 'ISO-8859-1//TRANSLIT//IGNORE', $string);
    // output: ABBASABAD
    // Yay! That's what I wanted!
?>

Be very conscientious with your character encodings, so you are keeping the same encoding at all stages in the process - front end, form submission, encoding of the source files. Default encoding in PHP and in forms is ISO-8859-1, before PHP 5.4 where it changed to be UTF8 (finally!).

There's a couple of functions you can play around with for ideas. First is from CakePHP's inflector class, called slug:

public static function slug($string, $replacement = '_') {
    $quotedReplacement = preg_quote($replacement, '/');

    $merge = array(
        '/[^\s\p{Ll}\p{Lm}\p{Lo}\p{Lt}\p{Lu}\p{Nd}]/mu' => ' ',
        '/\\s+/' => $replacement,
        sprintf('/^[%s]+|[%s]+$/', $quotedReplacement, $quotedReplacement) => '',
    );

    $map = self::$_transliteration + $merge;
    return preg_replace(array_keys($map), array_values($map), $string);
}

It depends on a self::$_transliteration array which is similar to what you were doing in your question - you can see the source for inflector on github.

Another is a function I use personally, which comes from here.

function slugify($text,$strict = false) {
    $text = html_entity_decode($text, ENT_QUOTES, 'UTF-8');
    // replace non letter or digits by -
    $text = preg_replace('~[^\\pL\d.]+~u', '-', $text);

    // trim
    $text = trim($text, '-');
    setlocale(LC_CTYPE, 'en_GB.utf8');
    // transliterate
    if (function_exists('iconv')) {
        $text = iconv('utf-8', 'us-ascii//TRANSLIT', $text);
    }

    // lowercase
    $text = strtolower($text);
    // remove unwanted characters
    $text = preg_replace('~[^-\w.]+~', '', $text);
    if (empty($text)) {
        return 'empty_$';
    }
    if ($strict) {
        $text = str_replace(".", "_", $text);
    }
    return $text;
}

What those functions do is transliterate and create 'slugs' from arbitrary text input, which is a very very useful thing to have in your toolchest when making web apps.

6
  • 2
    Thank you for answer, the problem was that I have not added " accept-charset="UTF-8" " in the form. Apr 14, 2012 at 11:10
  • welcome, i guess all those functions weren't really needed :) but may as well take a look.
    – danp
    Apr 14, 2012 at 11:14
  • 1
    I continued to use my function, because with it I can determine exactly with which letter I will replace some letter. Apr 14, 2012 at 14:46
  • Is there anyway keeping åäö?
    – embe
    Oct 27, 2016 at 10:30
  • @embe use iso-8859-1 instead of ascii maybe?
    – Svish
    Jan 13, 2017 at 5:23
41

Here is a way to have some flexibility in what should be discarded and what should be replaced. This is how I currently do it.

$string = 'À some string with junk Ĩ Ä ';

$replace = [
    '&lt;' => '', '&gt;' => '', '&#039;' => '', '&amp;' => '',
    '&quot;' => '', 'À' => 'A', 'Á' => 'A', 'Â' => 'A', 'Ã' => 'A', 'Ä' => 'Ae',
    '&Auml;' => 'A', 'Å' => 'A', 'Ā' => 'A', 'Ą' => 'A', 'Ă' => 'A', 'Æ' => 'Ae',
    'Ç' => 'C', 'Ć' => 'C', 'Č' => 'C', 'Ĉ' => 'C', 'Ċ' => 'C', 'Ď' => 'D', 'Đ' => 'D',
    'Ð' => 'D', 'È' => 'E', 'É' => 'E', 'Ê' => 'E', 'Ë' => 'E', 'Ē' => 'E',
    'Ę' => 'E', 'Ě' => 'E', 'Ĕ' => 'E', 'Ė' => 'E', 'Ĝ' => 'G', 'Ğ' => 'G',
    'Ġ' => 'G', 'Ģ' => 'G', 'Ĥ' => 'H', 'Ħ' => 'H', 'Ì' => 'I', 'Í' => 'I',
    'Î' => 'I', 'Ï' => 'I', 'Ī' => 'I', 'Ĩ' => 'I', 'Ĭ' => 'I', 'Į' => 'I',
    'İ' => 'I', 'IJ' => 'IJ', 'Ĵ' => 'J', 'Ķ' => 'K', 'Ł' => 'L', 'Ľ' => 'L',
    'Ĺ' => 'L', 'Ļ' => 'L', 'Ŀ' => 'L', 'Ñ' => 'N', 'Ń' => 'N', 'Ň' => 'N',
    'Ņ' => 'N', 'Ŋ' => 'N', 'Ò' => 'O', 'Ó' => 'O', 'Ô' => 'O', 'Õ' => 'O',
    'Ö' => 'Oe', '&Ouml;' => 'Oe', 'Ø' => 'O', 'Ō' => 'O', 'Ő' => 'O', 'Ŏ' => 'O',
    'Œ' => 'OE', 'Ŕ' => 'R', 'Ř' => 'R', 'Ŗ' => 'R', 'Ś' => 'S', 'Š' => 'S',
    'Ş' => 'S', 'Ŝ' => 'S', 'Ș' => 'S', 'Ť' => 'T', 'Ţ' => 'T', 'Ŧ' => 'T',
    'Ț' => 'T', 'Ù' => 'U', 'Ú' => 'U', 'Û' => 'U', 'Ü' => 'Ue', 'Ū' => 'U',
    '&Uuml;' => 'Ue', 'Ů' => 'U', 'Ű' => 'U', 'Ŭ' => 'U', 'Ũ' => 'U', 'Ų' => 'U',
    'Ŵ' => 'W', 'Ý' => 'Y', 'Ŷ' => 'Y', 'Ÿ' => 'Y', 'Ź' => 'Z', 'Ž' => 'Z',
    'Ż' => 'Z', 'Þ' => 'T', 'à' => 'a', 'á' => 'a', 'â' => 'a', 'ã' => 'a',
    'ä' => 'ae', '&auml;' => 'ae', 'å' => 'a', 'ā' => 'a', 'ą' => 'a', 'ă' => 'a',
    'æ' => 'ae', 'ç' => 'c', 'ć' => 'c', 'č' => 'c', 'ĉ' => 'c', 'ċ' => 'c',
    'ď' => 'd', 'đ' => 'd', 'ð' => 'd', 'è' => 'e', 'é' => 'e', 'ê' => 'e',
    'ë' => 'e', 'ē' => 'e', 'ę' => 'e', 'ě' => 'e', 'ĕ' => 'e', 'ė' => 'e',
    'ƒ' => 'f', 'ĝ' => 'g', 'ğ' => 'g', 'ġ' => 'g', 'ģ' => 'g', 'ĥ' => 'h',
    'ħ' => 'h', 'ì' => 'i', 'í' => 'i', 'î' => 'i', 'ï' => 'i', 'ī' => 'i',
    'ĩ' => 'i', 'ĭ' => 'i', 'į' => 'i', 'ı' => 'i', 'ij' => 'ij', 'ĵ' => 'j',
    'ķ' => 'k', 'ĸ' => 'k', 'ł' => 'l', 'ľ' => 'l', 'ĺ' => 'l', 'ļ' => 'l',
    'ŀ' => 'l', 'ñ' => 'n', 'ń' => 'n', 'ň' => 'n', 'ņ' => 'n', 'ʼn' => 'n',
    'ŋ' => 'n', 'ò' => 'o', 'ó' => 'o', 'ô' => 'o', 'õ' => 'o', 'ö' => 'oe',
    '&ouml;' => 'oe', 'ø' => 'o', 'ō' => 'o', 'ő' => 'o', 'ŏ' => 'o', 'œ' => 'oe',
    'ŕ' => 'r', 'ř' => 'r', 'ŗ' => 'r', 'š' => 's', 'ù' => 'u', 'ú' => 'u',
    'û' => 'u', 'ü' => 'ue', 'ū' => 'u', '&uuml;' => 'ue', 'ů' => 'u', 'ű' => 'u',
    'ŭ' => 'u', 'ũ' => 'u', 'ų' => 'u', 'ŵ' => 'w', 'ý' => 'y', 'ÿ' => 'y',
    'ŷ' => 'y', 'ž' => 'z', 'ż' => 'z', 'ź' => 'z', 'þ' => 't', 'ß' => 'ss',
    'ſ' => 'ss', 'ый' => 'iy', 'А' => 'A', 'Б' => 'B', 'В' => 'V', 'Г' => 'G',
    'Д' => 'D', 'Е' => 'E', 'Ё' => 'YO', 'Ж' => 'ZH', 'З' => 'Z', 'И' => 'I',
    'Й' => 'Y', 'К' => 'K', 'Л' => 'L', 'М' => 'M', 'Н' => 'N', 'О' => 'O',
    'П' => 'P', 'Р' => 'R', 'С' => 'S', 'Т' => 'T', 'У' => 'U', 'Ф' => 'F',
    'Х' => 'H', 'Ц' => 'C', 'Ч' => 'CH', 'Ш' => 'SH', 'Щ' => 'SCH', 'Ъ' => '',
    'Ы' => 'Y', 'Ь' => '', 'Э' => 'E', 'Ю' => 'YU', 'Я' => 'YA', 'а' => 'a',
    'б' => 'b', 'в' => 'v', 'г' => 'g', 'д' => 'd', 'е' => 'e', 'ё' => 'yo',
    'ж' => 'zh', 'з' => 'z', 'и' => 'i', 'й' => 'y', 'к' => 'k', 'л' => 'l',
    'м' => 'm', 'н' => 'n', 'о' => 'o', 'п' => 'p', 'р' => 'r', 'с' => 's',
    'т' => 't', 'у' => 'u', 'ф' => 'f', 'х' => 'h', 'ц' => 'c', 'ч' => 'ch',
    'ш' => 'sh', 'щ' => 'sch', 'ъ' => '', 'ы' => 'y', 'ь' => '', 'э' => 'e',
    'ю' => 'yu', 'я' => 'ya'
];

echo str_replace(array_keys($replace), $replace, $string);  
1
24

As of PHP >= 5.4.0

$translatedString = transliterator_transliterate('Any-Latin; Latin-ASCII; [\u0080-\u7fff] remove', $string);
0
9

The string $chain is in the same character encoding as the characters in the array - it's possible, even likely, that the $first_name string is in a different encoding, and so those characters don't match. You might want to try using the multibyte string functions instead.

Try mb_convert_encoding. You might also want to try using HTML_ENTITIES as the to_encoding parameter, then you don't need to worry about how the characters will get converted - it will be very predictable.

Assuming your input to this script is in UTF-8, probably not a bad place to start...

$first_name = mb_convert_encoding($first_name, "HTML-ENTITIES", "UTF-8"); 
2
  • I tried this and it's not. echo $first_name; // print áááááábéééééébšššš $first_name = mb_convert_encoding($first_name, "HTML-ENTITIES", "UTF-8"); echo $first_name; // print áááááábéééééébšššš Apr 14, 2012 at 10:58
  • Thank you for answer, the problem was that I have not added " accept-charset="UTF-8" " in the form. Apr 14, 2012 at 11:10
7

Wish I found this thread sooner. The function I made (that took me way too long) is below:

function CheckLetters($field){
    $letters = [
        0 => "a à á â ä æ ã å ā",
        1 => "c ç ć č",
        2 => "e é è ê ë ę ė ē",
        3 => "i ī į í ì ï î",
        4 => "l ł",
        5 => "n ñ ń",
        6 => "o ō ø œ õ ó ò ö ô",
        7 => "s ß ś š",
        8 => "u ū ú ù ü û",
        9 => "w ŵ",
        10 => "y ŷ ÿ",
        11 => "z ź ž ż",
    ];
    foreach ($letters as &$values){
        $newValue = substr($values, 0, 1);
        $values = substr($values, 2, strlen($values));
        $values = explode(" ", $values);
        foreach ($values as &$oldValue){
            while (strpos($field,$oldValue) !== false){
                $field = preg_replace("/" . $oldValue . '/', $newValue, $field, 1);
            }
        }
    }
    return $field;
}
6
  • 1
    This is a genius piece of code. Especially for searching databases that have inconsistent data. I use it to search a country database that is filled with user provided data. This really helps in not adding incorrectly written usercontent. A big thumbs up from me. Nov 9, 2016 at 22:15
  • Thanks for the feedback. Incorrectly entered user content is exactly why this was written :) Nov 18, 2016 at 7:56
  • just an sidenote, extra starting braces in beginning of the function should be removed obviously Jul 26, 2017 at 16:22
  • Nice piece of code, it does however not work if the first character is "dirty". Try CheckLetters('åäö');
    – sboss
    Nov 28, 2017 at 10:09
  • @sboss thanks for pointing that out. Was caused by while (strpos($field,$oldValue)) as strpos returns 0 if character found at index 0 of string. Fixed by doing a Not Identical comparison. Nov 29, 2017 at 7:42
5

Simple function. Transform strings like 'Ábç Éfg' to 'abc_efg'

/**
 * @param $str
 * @return mixed
 */
function sanitizeString($str) {
    $str = preg_replace('/[áàãâä]/ui', 'a', $str);
    $str = preg_replace('/[éèêë]/ui', 'e', $str);
    $str = preg_replace('/[íìîï]/ui', 'i', $str);
    $str = preg_replace('/[óòõôö]/ui', 'o', $str);
    $str = preg_replace('/[úùûü]/ui', 'u', $str);
    $str = preg_replace('/[ç]/ui', 'c', $str);
    $str = preg_replace('/[^a-z0-9]/i', '_', $str);
    $str = preg_replace('/_+/', '_', $str);

    return $str;
}
1
  • This doesn't keep the case.
    – Toto
    Aug 20, 2019 at 12:21
3

CodeIgniter way:

$this->load->helper('text');

$string = convert_accented_characters($string);

This function uses a companion config file application/config/foreign_chars.php to define the to and from array for transliteration.

https://www.codeigniter.com/user_guide/helpers/text_helper.html#ascii_to_entities

4
  • 1
    1- The question was asked more than 2 years ago 2- The question has an accepted answer. 3- The OP has not mentioned CI Framework and is using plain php so your answer has nothing to do with the question. please pay more attention to the questions before posting an answer
    – EhsanT
    Nov 19, 2016 at 5:08
  • 1
    This helped me after a day of frustration.. Really a good answer for Codeigniter Jun 20, 2017 at 6:20
  • 2
    @EhsanT 1. The age of the question is irrelevant. In fact, this site/resource benefits the most when people contribute to old content rather than waiting at the front door for duplicate questions to be posted. 2. An accepted answer doesn't bar anyone from posting a new answer. People should never be deterred from posting after the green tick is awarded. Though probably not appropriate for this case, the green tick is transferrable. 3. The OP doesn't need to post a whitelist or blacklist of libraries or frameworks. Future researchers may be using CI -- this answer will help them. Sep 20, 2019 at 22:29
  • Fair enough @mickmackusa .
    – EhsanT
    Sep 24, 2019 at 19:01
1
function correctedText($txt=''){
  $ss = str_split($txt);
    
  for($i=0; $i<count($ss); $i++){
    $asciiNumber = ord($ss[$i]);// get the ascii dec of a single character
    
    // asciiNumber will be from the DEC column showing at https://www.ascii-code.com
    
    // capital letters only checked 
    if($asciiNumber >= 192 && $asciiNumber <= 197)$ss[$i] = 'A';
    elseif($asciiNumber == 198)$ss[$i] = 'AE';
    elseif($asciiNumber == 199)$ss[$i] = 'C';
    elseif($asciiNumber >= 200 && $asciiNumber <= 203)$ss[$i] = 'E';
    elseif($asciiNumber >= 204 && $asciiNumber <= 207)$ss[$i] = 'I';
    elseif($asciiNumber == 209)$ss[$i] = 'N';
    elseif($asciiNumber >= 210 && $asciiNumber <= 214)$ss[$i] = 'O';
    elseif($asciiNumber == 216)$ss[$i] = 'O';
    elseif($asciiNumber >= 217 && $asciiNumber <= 220)$ss[$i] = 'U';
    elseif($asciiNumber == 221)$ss[$i] = 'Y';
  }
    
  $txt = implode('', $ss);
    
  return $txt;
}
0

No one notice that the searching array is with commas inside???

$pattern = array("'é'", "'è'", "'ë'", "'ê'", "'É'", "'È'", "'Ë'", "'Ê'", "'á'", "'à'", "'ä'", "'â'", "'å'", "'Á'", "'À'", "'Ä'", "'Â'", "'Å'", "'ó'", "'ò'", "'ö'", "'ô'", "'Ó'", "'Ò'", "'Ö'", "'Ô'", "'í'", "'ì'", "'ï'", "'î'", "'Í'", "'Ì'", "'Ï'", "'Î'", "'ú'", "'ù'", "'ü'", "'û'", "'Ú'", "'Ù'", "'Ü'", "'Û'", "'ý'", "'ÿ'", "'Ý'", "'ø'", "'Ø'", "'œ'", "'Œ'", "'Æ'", "'ç'", "'Ç'");

A simple replace with this one will 100% work! :)

$pattern = array("é", "è", "ë", "ê", "É", "È", "Ë", "Ê", "á", "à", "ä", "â", "å", "Á", "À", "Ä", "Â", "Å", "ó", "ò", "ö", "ô", "Ó", "Ò", "Ö", "Ô", "í", "ì", "ï", "î", "Í", "Ì", "Ï", "Î", "ú", "ù", "ü", "û", "Ú", "Ù", "Ü", "Û", "ý", "ÿ", "Ý", "ø", "Ø", "œ", "Œ", "Æ", "ç", "Ç");

str_replace is working perfectly! Just be more careful in future! It's a junior mistake.

1
  • 2
    The "commas' are single quotes or apostrophes. They are being used as pattern delimiters for the single character regular expression (a poor choice of delimiter in my opinion). str_replace() was in an earlier answer on this page 10 years ago. Feb 21 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.