Let's start with a slight variation of Ericson's answer and build more //TRANSLIT
features on it:
Decompose chars to gain ASCII-String
public class Translit {
private static final Charset US_ASCII = Charset.forName("US-ASCII");
private static String toAscii(final String input) {
final CharsetEncoder charsetEncoder = US_ASCII.newEncoder();
final char[] decomposed = Normalizer.normalize(input, Normalizer.Form.NFKD).toCharArray();
final StringBuilder sb = new StringBuilder(decomposed.length);
for (int i = 0; i < decomposed.length; ) {
final int codePoint = Character.codePointAt(decomposed, i);
final int charCount = Character.charCount(codePoint);
if(charsetEncoder.canEncode(CharBuffer.wrap(decomposed, i, charCount))) {
sb.append(decomposed, i, charCount);
}
i += charCount;
}
return sb.toString();
}
public static void main(String[] args) {
final String a = "Michèleäöüß";
System.out.println(a + " => " + toAscii(a));
System.out.println(a.toUpperCase() + " => " + toAscii(a.toUpperCase()));
}
}
While this should behave the same for US-ASCII this solution is easier to adopt for different target encodings. (As characters are decomposed first this does not necessarily yield better results for other encodings though)
The function is safe for supplementary code points (which is a bit overkill for ASCII as target, but may reduce head-aches if another target encoding is chosen).
Also note, that a regular Java-String is returned; if you need an ASCII-byte[]
you still need to convert it (but as we ensured there are no offending characters...).
And this is how you could extend it to more character-sets:
Replace or decompose characters to gain a String
encodeable in supplied Charset
import java.nio.CharBuffer;
import java.nio.charset.Charset;
import java.nio.charset.CharsetEncoder;
import java.text.Normalizer;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
/**
* Created for http://stackoverflow.com/a/22841035/1266906
*/
public class Translit {
public static final Charset US_ASCII = Charset.forName("US-ASCII");
public static final Charset ISO_8859_1 = Charset.forName("ISO-8859-1");
public static final Charset UTF_8 = Charset.forName("UTF-8");
public static final HashMap<Integer, String> REPLACEMENTS = new ReplacementBuilder().put('„', '"')
.put('“', '"')
.put('”', '"')
.put('″', '"')
.put('€', "EUR")
.put('ß', "ss")
.put('•', '*')
.getMap();
private static String toCharset(final String input, Charset charset) {
return toCharset(input, charset, Collections.<Integer, String>emptyMap());
}
private static String toCharset(final String input,
Charset charset,
Map<? super Integer, ? extends String> replacements) {
final CharsetEncoder charsetEncoder = charset.newEncoder();
return toCharset(input, charsetEncoder, replacements);
}
private static String toCharset(String input,
CharsetEncoder charsetEncoder,
Map<? super Integer, ? extends String> replacements) {
char[] data = input.toCharArray();
final StringBuilder sb = new StringBuilder(data.length);
for (int i = 0; i < data.length; ) {
final int codePoint = Character.codePointAt(data, i);
final int charCount = Character.charCount(codePoint);
CharBuffer charBuffer = CharBuffer.wrap(data, i, charCount);
if (charsetEncoder.canEncode(charBuffer)) {
sb.append(data, i, charCount);
} else if (replacements.containsKey(codePoint)) {
sb.append(toCharset(replacements.get(codePoint), charsetEncoder, replacements));
} else {
// Only perform NFKD Normalization after ensuring the original character is invalid as this is a irreversible process
final char[] decomposed = Normalizer.normalize(charBuffer, Normalizer.Form.NFKD).toCharArray();
for (int j = 0; j < decomposed.length; ) {
int decomposedCodePoint = Character.codePointAt(decomposed, j);
int decomposedCharCount = Character.charCount(decomposedCodePoint);
if (charsetEncoder.canEncode(CharBuffer.wrap(decomposed, j, decomposedCharCount))) {
sb.append(decomposed, j, decomposedCharCount);
} else if (replacements.containsKey(decomposedCodePoint)) {
sb.append(toCharset(replacements.get(decomposedCodePoint), charsetEncoder, replacements));
}
j += decomposedCharCount;
}
}
i += charCount;
}
return sb.toString();
}
public static void main(String[] args) {
final String a = "Michèleäöü߀„“”″•";
System.out.println(a + " => " + toCharset(a, US_ASCII));
System.out.println(a + " => " + toCharset(a, ISO_8859_1));
System.out.println(a + " => " + toCharset(a, UTF_8));
System.out.println(a + " => " + toCharset(a, US_ASCII, REPLACEMENTS));
System.out.println(a + " => " + toCharset(a, ISO_8859_1, REPLACEMENTS));
System.out.println(a + " => " + toCharset(a, UTF_8, REPLACEMENTS));
}
public static class MapBuilder<K, V> {
private final HashMap<K, V> map;
public MapBuilder() {
map = new HashMap<K, V>();
}
public MapBuilder<K, V> put(K key, V value) {
map.put(key, value);
return this;
}
public HashMap<K, V> getMap() {
return map;
}
}
public static class ReplacementBuilder extends MapBuilder<Integer, String> {
public ReplacementBuilder() {
super();
}
@Override
public ReplacementBuilder put(Integer input, String replacement) {
super.put(input, replacement);
return this;
}
public ReplacementBuilder put(Integer input, char replacement) {
return this.put(input, String.valueOf(replacement));
}
public ReplacementBuilder put(char input, String replacement) {
return this.put((int) input, replacement);
}
public ReplacementBuilder put(char input, char replacement) {
return this.put((int) input, String.valueOf(replacement));
}
}
}
I would strongly recommend building an extensive replacement-table as the simple example already shows how you otherwise might lose desired information like €
. For ASCII this implementation is of course a bit slower as decomposition is only done on demand and the StringBuilder
now may need to grow to hold the replacements.
GNU's iconv uses the replacements listed in translit.def to perform a //TRANSLIT
-conversion and you can use a method like this if you want to use it as replacement-map:
Import original //TRANSLIT
-replacements
private static Map<Integer, String> readReplacements() {
HashMap<Integer, String> map = new HashMap<>();
InputStream stream = Translit.class.getResourceAsStream("/translit.def");
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(stream, UTF_8));
Pattern pattern = Pattern.compile("^([0-9A-Fa-f]+)\t(.?[^\t]*)\t#(.*)$");
try {
String line;
while ((line = bufferedReader.readLine()) != null) {
if (line.charAt(0) != '#') {
Matcher matcher = pattern.matcher(line);
if (matcher.find()) {
map.put(Integer.valueOf(matcher.group(1), 16), matcher.group(2));
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
return map;
}