0

I was trying to tokenize an input file from sentences into tokens(words). For example, "This is a test file." into five words "this" "is" "a" "test" "file", omitting the punctuations and the white spaces. And store them into an arraylist. I tried to write some codes like this:

public static ArrayList<String> tokenizeFile(File in) throws IOException {
    String strLine;
    String[] tokens;
    //create a new ArrayList to store tokens
    ArrayList<String> tokenList = new ArrayList<String>();

    if (null == in) {
        return tokenList;
    } else {
        FileInputStream fStream = new FileInputStream(in);
        DataInputStream dataIn = new DataInputStream(fStream);
        BufferedReader br = new BufferedReader(new InputStreamReader(dataIn));

        while (null != (strLine = br.readLine())) {
            if (strLine.trim().length() != 0) {

                //make sure strings are independent of capitalization and then tokenize them
                strLine = strLine.toLowerCase();

                //create regular expression pattern to split
                //first letter to be alphabetic and the remaining characters to be alphanumeric or '
                String pattern = "^[A-Za-z][A-Za-z0-9'-]*$";
                tokens = strLine.split(pattern);
                int tokenLen = tokens.length;

                for (int i = 1; i <= tokenLen; i++) {
                    tokenList.add(tokens[i - 1]);
                }
            }
        }
        br.close();
        dataIn.close();
    }
    return tokenList;
}

This code works fine except I found out that instead of make a whole file into several words(tokens), it made a whole line into a token. "area area" becomes a token, instead of "area" appeared twice. I don't see the error in my codes. I believe maybe it's something wrong with my trim(). Any valuable advices is appreciated. Thank you so much.

Maybe I should use scanner instead?? I'm confused.

2 Answers 2

1

I think Scanner is more approprate for this task. As to this code, you should fix regex, try "\\s+";

1
  • Thank you!! Now I know my regular expression is something unusual.
    – Frank
    Jan 18, 2016 at 6:11
1

Try pattern as String pattern = "[^\\w]"; in the same code

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.