0

I have two versions of the same document (D, say) containing multilingual text (English and others):

I. One is encoded in ASCII with Unicode code-points represented as character entity references (i.e. Unicode characters are of the form &#N, where N is the decimal equivalent of the Unicode hex value)

II. The other is UTF-8 encoding.

Q 1:

I have a separate list of words (encoded in UTF-8, and in more than one language), that I have to remove from the document D. How should I proceed?

Can I use regex to clean D? For doc type I, I believe I have to specify the whole &#N patterns for each word in the list when I form the regex.

Should the task be easier for doc type II, now that I can specify the non-English characters directly in the regex (my emacs is configured to use these non-English fonts) ?

Q 2:

I have a huge collections of such document D's. What should be the best algorithm to remove words from each of these documents? A table look-up is straight-forward but probably the slowest. Should I regex through each?

2
  • It would be helpful to provide a list of possible languages. As described, your requirements are difficult if not impossible to fulfill for every language. Dec 3, 2010 at 7:43
  • The text is in English and Bengali.
    – rup
    Dec 5, 2010 at 4:44

1 Answer 1

1

I suggest processing the entities first so that the two sorts of files look the same. When you’re done removing, put the first set back into their encoded form.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.