18

I have inherited a poorly written web application that seems to have errors when it tries to read in an xml document stored in the database that has an "&" in it. For example there will be a tag with the contents: "Prepaid & Charge". Is there some secret simple thing to do to have it not get an error parsing that character, or am I missing something obvious?

EDIT: Are there any other characters that will cause this same type of parser error for not being well formed?

6 Answers 6

43

The problem is the xml is not well-formed. Properly generated xml would list the data like this:

Prepaid & Charge

I've fixed the same problem before, and I did it with this regex:

Regex badAmpersand = new Regex("&(?![a-zA-Z]{2,6};|#[0-9]{2,4};)");

Combine that with a string constant defined like this:

const string goodAmpersand = "&";

Now you can say badAmpersand.Replace(<your input>, goodAmpersand);

Note a simple String.Replace("&", "&amp;") isn't good enough, since you can't know in advance for a given document whether any & characters will be coded correctly, incorrectly, or even both in the same document.

The catches here are you have to do this to your xml document before loading it into your parser, which likely means an extra pass through the document. Also, it does not account for ampersands inside of a CDATA section. Finally, it only catches ampersands, not other illegal characters like <. Update: based on the comment, I need to update the expression for hex-coded (&#x...;) entities as well.

Regarding which characters can cause problems, the actual rules are a little complex. For example, certain characters are allowed in data, but not as the first letter of an element name. And there's no simple list of illegal characters. Instead, large (non-contiguous) swaths of UNICODE are defined as legal, and anything outside that is illegal.

When it comes down to it, you have to trust your document source to have at least a certain amount of compliance and consistency. For example, I've found people are often smart enough to make sure the tags work properly and escape <, even if they don't know that & isn't allowed, hence your problem today. However, the best thing would be to get this fixed at the source.

Oh, and a note about the CDATA suggestion: I use that to make sure xml I'm creating is well-formed, but when dealing with existing xml from outside, I find the regex method easier.

1
  • Hex-coded &#x...; character references might also be present. I suggest trying to parse the document normally first, any only try to apply fixup hacks if a well-formedness error results. Then you can be sure future fixed valid documents will always work properly.
    – bobince
    Feb 9, 2009 at 17:02
4

The web application isn't at fault, the XML document is. Ampersands in XML should be encoded as &amp;. Failure to do so is a syntax error.

Edit: in answer to the followup question, yes there are all kinds of similar errors. For example, unbalanced tags, unencoded less-than signs, unquoted attribute values, octets outside of the character encoding and various Unicode oddities, unrecognised entity references, and so on. In order to get any decent XML parser to consume a document, that document must be well-formed. The XML specification requires that a parser encountering a malformed document throw a fatal error.

2
  • So, you think I can lace the blame on my lazy integrations team for sending bad XML huh? Something tells me I still get to fix it on my side. :-( Sep 23, 2008 at 14:57
  • If at all possible, yes, fix the problem at the source. If you absolutely must deal with a bad source that won't fix their system, then attempt to fix it up before it goes in the database.
    – Jim
    Sep 23, 2008 at 15:01
4

The other answers are all correct, and I concur with their advice, but let me just add one thing:

PLEASE do not make applications that work with non well-formed XML, it just makes the rest of our lives more difficult :).

Granted, there are times when you really just don't have a choice if you have no control over the other end, but you should really have it throwing a fatal error and complaining very loudly and explicitly about what is broken when such an event occurs.

You could probably take it one step further and say "Ack! This XML is broken in these places and for these reasons, here's how I tried to fix it to make it well-formed: ...".

I'm not overly familiar with the MSXML APIs, but most good XML parsers will allow you to install error handlers so that you can trap the exact line/column number where errors are appearing along with getting the error code and message.

2
  • I tend to agree. My first course of action is complaining and I'll see if they will fix it. Problem is it's a 2 min fix on my end and much more on their end. Thanks for the advice in any event. Sep 23, 2008 at 15:53
  • +1. If you fix it up, at least continue to raise warnings. Continue doing it ‘the wrong way’ and you'll surely eventually meet data that breaks whatever you're doing to fix it.
    – bobince
    Feb 9, 2009 at 16:59
3

Your database doesn't contain XML documents. It contains some well-formed XML documents and some strings that look like XML to a human.

If it's at all possible, you should fix this - in particular, you should fix whatever process is generating the malformed XML documents. Fixing the program that reads data out of this database is just putting wallpaper over a crack in the wall.

2

You can replace & with &amp;

Or you might also be able to use CDATA sections.

1
  • 1
    Globally encoding & will lose any real entity references in the data.
    – bobince
    Feb 9, 2009 at 16:57
2

There are several characters which will cause XML data to be reported as badly-formed.

From w3schools:

Characters like "<" and "&" are illegal in XML elements.

The best solution for input you can't trust to be XML-compliant is to wrap it in CDATA tags, e.g.

<![CDATA[This is my wonderful & great user text]]>

Everything within the <![CDATA[ and ]]> tags is ignored by the parser.

3
  • Perhaps a clarification of 'everything within CDATA tags apart from the end tag is ignored by the parser?
    – workmad3
    Sep 23, 2008 at 15:04
  • Not quite everything.. ]]> isn't. Which means if he's sent an xml that has a CDATA section in it, or even just a ]]> chunk of text, it'll break.
    – davenpcj
    Sep 23, 2008 at 15:06
  • If you do encounter a ]]> in your input, split your input in 2 CDATA sections: One that ends in ]], and one that starts with >. The parser will reconcatenate the two parts again.
    – MSalters
    Oct 17, 2008 at 14:17

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.