288

I am working with some XML that holds strings like:

<node>This is a string</node>

Some of the strings that I am passing to the nodes will have characters like &, #, $, etc.:

<node>This is a string & so is this</node>

This is not valid due to &.

I cannot wrap these strings in CDATA as they need to be as they are. I tried looking for a list of characters that cannot be put in XML nodes without being in a CDATA.

Can someone point me in the direction of one or provide me with a list of illegal characters?

3
  • 4
    Any valid reason for not using CDATA? Apr 8, 2009 at 14:00
  • 2
    Yes, I am passing the string to a CMS called Fatwire and the node with the data cannot be in a CDATA, i'm not sure why it's the way Fatwire works :(
    – RailsSon
    Apr 8, 2009 at 14:07
  • @Peter: How can I use CDATA in my case? stackoverflow.com/questions/6906705/…
    – Radek
    Aug 2, 2011 at 3:02

16 Answers 16

298
+100

OK, let's separate the question of the characters that:

  1. aren't valid at all in any XML document.
  2. need to be escaped.

The answer provided by @dolmen in "https://stackoverflow.com/questions/730133/invalid-characters-in-xml/5110103#5110103" is still valid but needs to be updated with the XML 1.1 specification.

1. Invalid characters

The characters described here are all the characters that are allowed to be inserted in an XML document.

1.1. In XML 1.0

The global list of allowed characters is:

[2] Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

Basically, the control characters and characters out of the Unicode ranges are not allowed. This means also that calling for example the character entity &#x3; is forbidden.

1.2. In XML 1.1

The global list of allowed characters is:

[2] Char ::= [#x1-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

[2a] RestrictedChar ::= [#x1-#x8] | [#xB-#xC] | [#xE-#x1F] | [#x7F-#x84] | [#x86-#x9F]

This revision of the XML recommendation has extended the allowed characters so control characters are allowed, and takes into account a new revision of the Unicode standard, but these ones are still not allowed : NUL (x00), xFFFE, xFFFF...

However, the use of control characters and undefined Unicode char is discouraged.

It can also be noticed that all parsers do not always take this into account and XML documents with control characters may be rejected.

2. Characters that need to be escaped (to obtain a well-formed document):

The < must be escaped with a &lt; entity, since it is assumed to be the beginning of a tag.

The & must be escaped with a &amp; entity, since it is assumed to be the beginning a entity reference

The > should be escaped with &gt; entity. It is not mandatory -- it depends on the context -- but it is strongly advised to escape it.

The ' should be escaped with a &apos; entity -- mandatory in attributes defined within single quotes but it is strongly advised to always escape it.

The " should be escaped with a &quot; entity -- mandatory in attributes defined within double quotes but it is strongly advised to always escape it.

4
  • 3
    " but it is strongly advised to always escape it" - Could you clarify that bit? Who advises that, and why? (The way I see it, there's nothing wrong with using literal quotes wherever they are syntactically allowed.)
    – Tomalak
    Jan 12, 2021 at 15:30
  • Shouldn't ' be escaped as &apos; instead ? w3.org/TR/REC-xml/#syntax
    – Simon
    Jan 24, 2022 at 10:08
  • 1
    @Simon hey, I didn't notice the answer has been modified because I originally wrote to escape with &apos;. However both will work since numeric character reference are equally recognized w3.org/TR/REC-xml/#dt-charref
    – potame
    Jan 24, 2022 at 10:41
  • 3
    For 2.: see stackoverflow.com/questions/1091945/… for details. These 5 characters needn't always be escaped, just in some circumstances. Mar 8, 2022 at 12:48
179

The list of valid characters is in the XML specification:

Char       ::=      #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]  /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
7
  • 8
    You should note that although they are legal characters, & < > " ' have to be escaped in certain contexts.
    – D.Shawley
    May 8, 2011 at 19:49
  • 7
    "Legal" in this context means that their final decoded values are legal, not that they are legal in the stream. As above, some legal values have to be escaped in-stream. Jul 16, 2011 at 1:59
  • I have an issue where 0x1c is an illegal character... Looking for a possibility in java how to avoid these....
    – basZero
    Dec 10, 2013 at 9:20
  • A nice overview which characters are valid and which are not can be found here validchar.com/d/xml10/xml10_namestart Feb 21, 2014 at 21:58
  • 8
    @xamde That list is nice, but it only shows the characters that may be used to start an XML element. The issue at hand is which characters are valid in an XML file in general. There are certain characters that are not allowed anywhere. Jun 23, 2014 at 19:58
166

The only illegal characters are &, < and > (as well as " or ' in attributes, depending on which character is used to delimit the attribute value: attr="must use &quot; here, ' is allowed" and attr='must use &apos; here, " is allowed').

They're escaped using XML entities, in this case you want &amp; for &.

Really, though, you should use a tool or library that writes XML for you and abstracts this kind of thing away for you so you don't have to worry about it.

13
  • 95
    Some controls characters are also not allowed. See my answer below.
    – dolmen
    Feb 24, 2011 at 20:36
  • 54
    Actually that's not quite true. A number of lower ascii characters are invalid also. If you try to write 0x03 to an Xml document you get an error typically and if you do manage to properly escape it into an XML document, most viewers will complain about the invalid character. Edge case but it does happen. Jan 2, 2012 at 9:56
  • 24
    This answer is absolutely wrong. Here is my XML exception with 0x12 illegal character 'System.Xml.XmlException: '', hexadecimal value 0x12, is an invalid character'
    – George
    Sep 30, 2014 at 15:54
  • 9
    It's also wrong in the other direction; as well as missing every single illegal character, the characters it does claim are illegal are perfectly legal, albeit with special meaning in the context.
    – Jon Hanna
    Dec 16, 2014 at 14:47
  • 8
    In XML 1.0 there are many illegal characters. In fact even using a character entity for most control characters will cause an error when parsing.
    – Thayne
    Nov 17, 2015 at 16:21
64

This is a C# code to remove the XML invalid characters from a string and return a new valid string.

public static string CleanInvalidXmlChars(string text) 
{ 
    // From xml spec valid chars: 
    // #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]     
    // any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. 
    string re = @"[^\x09\x0A\x0D\x20-\uD7FF\uE000-\uFFFD\u10000-\u10FFFF]"; 
    return Regex.Replace(text, re, ""); 
}
7
  • 6
    For Java, the regex pattern would be the same. And then you can use the method called replaceAll in the class String that expects a regex pattern as parameter. Check this: docs.oracle.com/javase/6/docs/api/java/lang/… Dec 10, 2013 at 14:37
  • 3
    I have such invalid characters in my string: SUSITARIMO D&#x5;L DARBO SUTARTIES This code doesn't remove &#x5; So the xml document fails to init. Jul 30, 2015 at 10:05
  • 2
    I believe you cannot just put this pattern into a .NET regex constructor. I don't think it recognizes \u10000 and \u10FFFF as single characters as they require two utf-16 char instances each, and according to the docs there might not be more that 4 digits. [\u10000-\u10FFFF] is most likely parsed as [\u1000, 0-\u10FF, F, F] which is weird looking but legal.
    – GSerg
    May 23, 2018 at 16:16
  • A better implementation that takes care of the utf-16 characters can be found here: stackoverflow.com/a/17735649/1639057 Oct 1, 2020 at 8:56
  • be careful to use this method, your valid UTF character will also be replaced with empty string, causing unexpected result on application
    – user991513
    Oct 19, 2020 at 15:11
18

The predeclared characters are:

& < > " '

See "What are the special characters in XML?" for more information.

1
  • Wrong. These are not all invalid. Only & and < are always invalid in the text.
    – rghome
    Oct 24, 2018 at 13:51
12

In addition to potame's answer, if you do want to escape using a CDATA block.

If you put your text in a CDATA block then you don't need to use escaping. In that case you can use all characters in the following range:

graphical representation of possible characters

Note: On top of that, you're not allowed to use the ]]> character sequence. Because it would match the end of the CDATA block.

If there are still invalid characters (e.g. control characters), then probably it's better to use some kind of encoding (e.g. base64).

2
  • 3
    Wether in a CDATA block or not, some characters are forbidden in XML.
    – dolmen
    Feb 9, 2017 at 16:45
  • 7
    exactly, isn't that what I wrote ? quote: "all characters in the following range". By which I mean, only the characters in this specific range. Other characters are not allowed. - fully agree ; but I don't understand the downvote. - no hard feelings though.
    – bvdb
    Feb 9, 2017 at 20:11
9

Another way to remove incorrect XML chars in C# is using XmlConvert.IsXmlChar (Available since .NET Framework 4.0)

public static string RemoveInvalidXmlChars(string content)
{
   return new string(content.Where(ch => System.Xml.XmlConvert.IsXmlChar(ch)).ToArray());
}

or you may check that all characters are XML-valid:

public static bool CheckValidXmlChars(string content)
{
   return content.All(ch => System.Xml.XmlConvert.IsXmlChar(ch));
}

.Net Fiddle

For example, the vertical tab symbol (\v) is not valid for XML, it is valid UTF-8, but not valid XML 1.0, and even many libraries (including libxml2) miss it and silently output invalid XML.

6

Another easy way to escape potentially unwanted XML / XHTML chars in C# is:

WebUtility.HtmlEncode(stringWithStrangeChars)
3
  • Invalid characters
    – dolmen
    Jul 21, 2017 at 22:35
  • 2
    He wrote Xml not Html.
    – Emanuele
    May 25, 2020 at 9:18
  • This works with < > & " ' but not control characters
    – nimblebit
    Oct 17 at 14:48
3

For Java folks, Apache has a utility class (StringEscapeUtils) that has a helper method escapeXml which can be used for escaping characters in a string using XML entities.

3

"XmlWriter and lower ASCII characters" worked for me

string code = Regex.Replace(item.Code, @"[\u0000-\u0008,\u000B,\u000C,\u000E-\u001F]", "");
2

In summary, valid characters in the text are:

  • tab, line-feed and carriage-return.
  • all non-control characters are valid except & and <.
  • > is not valid if following ]].

Sections 2.2 and 2.4 of the XML specification provide the answer in detail:

Characters

Legal characters are tab, carriage return, line feed, and the legal characters of Unicode and ISO/IEC 10646

Character data

The ampersand character (&) and the left angle bracket (<) must not appear in their literal form, except when used as markup delimiters, or within a comment, a processing instruction, or a CDATA section. If they are needed elsewhere, they must be escaped using either numeric character references or the strings " & " and " < " respectively. The right angle bracket (>) may be represented using the string " > ", and must, for compatibility, be escaped using either " > " or a character reference when it appears in the string " ]]> " in content, when that string is not marking the end of a CDATA section.

1
ampersand (&) is escaped to &amp;

double quotes (") are escaped to &quot;

single quotes (') are escaped to &apos; 

less than (<) is escaped to &lt; 

greater than (>) is escaped to &gt;

In C#, use System.Security.SecurityElement.Escape or System.Net.WebUtility.HtmlEncode to escape these illegal characters.

string xml = "<node>it's my \"node\" & i like it 0x12 x09 x0A  0x09 0x0A <node>";
string encodedXml1 = System.Security.SecurityElement.Escape(xml);
string encodedXml2= System.Net.WebUtility.HtmlEncode(xml);


encodedXml1
"&lt;node&gt;it&apos;s my &quot;node&quot; &amp; i like it 0x12 x09 x0A  0x09 0x0A &lt;node&gt;"

encodedXml2
"&lt;node&gt;it&#39;s my &quot;node&quot; &amp; i like it 0x12 x09 x0A  0x09 0x0A &lt;node&gt;"
1

In the Woodstox XML processor, invalid characters are classified by this code:

if (c == 0) {
    throw new IOException("Invalid null character in text to output");
}
if (c < ' ' || (c >= 0x7F && c <= 0x9F)) {
    String msg = "Invalid white space character (0x" + Integer.toHexString(c) + ") in text to output";
    if (mXml11) {
        msg += " (can only be output using character entity)";
    }
    throw new IOException(msg);
}
if (c > 0x10FFFF) {
    throw new IOException("Illegal unicode character point (0x" + Integer.toHexString(c) + ") to output; max is 0x10FFFF as per RFC");
}
/*
 * Surrogate pair in non-quotable (not text or attribute value) content, and non-unicode encoding (ISO-8859-x,
 * Ascii)?
 */
if (c >= SURR1_FIRST && c <= SURR2_LAST) {
    throw new IOException("Illegal surrogate pair -- can only be output via character entities, which are not allowed in this content");
}
throw new IOException("Invalid XML character (0x"+Integer.toHexString(c)+") in text to output");

Source from here

0

Remove invalid characters(restricted + discouraged) in xml document with Java

I had some trouble making a pattern from the xml 1.1 specification because it has characters that exceed 16 bits.

The issue with Java is, a char takes always 16 bits which means after the 64K first code points of Unicode, (i.e. range 0x0000 to 0xFFFF), the code points are non-BMP ("rare") Unicode character and can't be expressed by \u literal. Luckily starting with Java 7, we can use \x{foo} where foo is the hexadecimal representation of the code point.

Finally the below code removes all Restricted Characters + Discouraged Characters from a text.

static Pattern XMLCharInvalidPattern =
      Pattern.compile(
          "[\\x{1}-\\x{8}]|[\\x{B}-\\x{C}]|[\\x{E}-\\x{1F}]|[\\x{7F}-\\x{84}]|[\\x{86}-\\x{9F}]|[\\x{FDD0}-\\x{FDDF}]|[\\x{1FFFE}-\\x{1FFFF}]|[\\x{2FFFE}-\\x{2FFFF}]|[\\x{3FFFE}-\\x{3FFFF}]|[\\x{4FFFE}-\\x{4FFFF}]|[\\x{5FFFE}-\\x{5FFFF}]|[\\x{6FFFE}-\\x{6FFFF}]|[\\x{7FFFE}-\\x{7FFFF}]|[\\x{8FFFE}-\\x{8FFFF}]|[\\x{9FFFE}-\\x{9FFFF}]|[\\x{AFFFE}-\\x{AFFFF}]|[\\x{BFFFE}-\\x{BFFFF}]|[\\x{CFFFE}-\\x{CFFFF}]|[\\x{DFFFE}-\\x{DFFFF}]|[\\x{EFFFE}-\\x{EFFFF}]|[\\x{FFFFE}-\\x{FFFFF}]|[\\x{10FFFE}-\\x{10FFFF}]");

String invalidXmlText = "he\u0001ll\u0003o wo\uFDD0rl\u0084d";

String cleanXmlText = XMLCharInvalidPattern.matcher(invalidXmlText).replaceAll("");

// cleanXmlText = hello world
-2

Anyone tried this System.Security.SecurityElement.Escape(yourstring)? This will replace invalid XML characters in a string with their valid equivalent.

0
-5

For XSL (on really lazy days) I use:

capture="&amp;(?!amp;)" capturereplace="&amp;amp;"

to translate all &-signs that aren't follwed på amp; to proper ones.

We have cases where the input is in CDATA but the system which uses the XML doesn't take it into account. It's a sloppy fix, beware...

1
  • 11
    If it's sloppy, is it really necessary to post it here?
    – dolmen
    Mar 9, 2015 at 22:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.