13

I develop C++ cross platform using Microsoft Visual Studio on Windows and GCC on Ubuntu Linux.

In Visual Studio, I can use Unicode symbols like "π" and "²" in my code. Visual Studio always saves the source files as UTF-8 with BOM (Byte Order Mark).

For example:

// A = π.r²
double π = 3.14;

GCC happily compiles these files only if I remove the BOM first. If I do not remove the BOM, I get errors like these:

wwga_hydutils.cpp:28:9: error: stray ‘\317’ in program

wwga_hydutils.cpp:28:9: error: stray ‘\200’ in program

Which brings me to the question:

Is there a way to get GCC to compile UTF-8 files without first removing the BOM?


I'm using:

and:


As the first commenter pointed out, my problem was not the BOM, but having non-ASCII characters outside of string constants. GCC does not like non-ASCII characters in symbol names, but it turns out GCC is fully compatible with UTF-8 with BOM.

6
  • 2
    Works fine for me in gcc 4.4.5, using a string containing both the UNICODE characters in your question. File with BOM. Also, the error you get has nothing to do with the BOM, but seems to be that the UNICODE characters in question is outside any string (thats why they are called stray.) Oct 26, 2011 at 8:25
  • @JoachimPileborg yes the unicode characters are outside of the string, the "π" I was using as a symbol name, the "²" was just in comments. When I remove the BOM, it does eliminate the error from the console output, but I guess that's no gaurantee that GCC is really handling the characters how I expect.
    – Boinst
    Oct 26, 2011 at 15:35
  • 1
    It is an error to have a BOM in a UTF-8 stream, because it precluded catting three of them together and getting the correct result.
    – tchrist
    Oct 31, 2011 at 1:27
  • 2
    double π = 3.14; : typography +1, math -1.
    – chqrlie
    Jan 27, 2016 at 0:23
  • 1
    clang supports these symbols in identifiers, gcc only supports in strings,To use Λ (greek lambda) in identifiers in gcc use universal character name (ibm.com/support/knowledgecenter/en/ssw_ibm_i_74/rzarg/…), so a function funΛ(), would be written as fun\u039B() to be able to run in gcc. I changed my compiler to clang, and things worked fine. gcc's -finput-charset=UTF-8 -fextended-identifiers don't help either. -fextended-identifiers is simply for supporting universal character name format, if turn off(-fno-extended-identifiers) even fun\u039B() fails. Jul 16, 2019 at 6:10

2 Answers 2

4

According to the GCC Wiki, this isn't supported yet. You can use -fextended-identifiers and pre-process your code to convert the identifiers to UCN. From the linked page:

perl -pe 'BEGIN { binmode STDIN, ":utf8"; } s/(.)/ord($1) < 128 ? $1 : sprintf("\\U%08x", ord($1))/ge;' 

See also g++ unicode variable name and Unicode Identifiers and Source Code in C++11?

1
4

While Unicode identifiers are supported in GCC, UTF-8 input is not. Therefore, Unicode identifiers have to be encoded using \uXXXX and \UXXXXXXXX escape codes. However, a simple one-line patch to the C++ preprocessor allows GCC and g++ to process UTF-8 input provided a recent version of iconv that support C99 conversions is also installed. Details are present at UTF-8 Identifiers in GCC.

However, the patch is so simple it can be given right here:

diff -cNr gcc-5.2.0/libcpp/charset.c gcc-5.2.0-ejo/libcpp/charset.c

Output:

*** gcc-5.2.0/libcpp/charset.c  Mon Jan  5 04:33:28 2015
--- gcc-5.2.0-ejo/libcpp/charset.c  Wed Aug 12 14:34:23 2015
***************
*** 1711,1717 ****
    struct _cpp_strbuf to;
    unsigned char *buffer;

!   input_cset = init_iconv_desc (pfile, SOURCE_CHARSET, input_charset);
    if (input_cset.func == convert_no_conversion)
      {
        to.text = input;
--- 1711,1717 ----
    struct _cpp_strbuf to;
    unsigned char *buffer;

!   input_cset = init_iconv_desc (pfile, "C99", input_charset);
    if (input_cset.func == convert_no_conversion)
      {
        to.text = input;

Even with the patch, two command line options (-finput-charset and -fextended-identifiers) are needed to enable UTF-8 input. In particular, try something like

/usr/local/gcc-5.2/bin/gcc \
    -finput-charset=UTF-8 -fextended-identifiers \
    -o circle circle.c

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.