0

I would like to make a very simple lexical analyzer but I am a bit stuck. Let's say I have only two arithmetic operations (addition and multiplication) and two integers. An addition or a multiplication is made with the two operands (the order does not matter since it's an addition or multiplication). For example as input "ADD 4 8", is expected 4+8. Below is what I've done so far. I can extract the operands, but curiously not the operator either "+" or "*". As stated, I don't really know how to go forward, and perform a simple add or mul.Thanks for your help.

private static enum Type {
    ADD, MUL, OPERAND
}

private static class Token<TokenType, TokenValue> {
    private final TokenType type;
    private final TokenValue value;

    public Token(TokenType t, TokenValue value) {
        this.type = t;
        this.value = value;
    }

    public String toString() {
        return "Token value: " + this.value + "\n" + "Token type: " + this.type;
    }
}

private static String getOperand(String operand, int index) {

    int i = index;
    while (i < operand.length()) {
        if (Character.isDigit(operand.charAt(i))) {
            i++;
        } else {
            return operand.substring(index, i);
        }
    }
    return operand.substring(index, i);
}

private static ArrayList<Token<Type, String>> lex(String expression) {

    ArrayList<Token<Type, String>> tokens = new ArrayList<>();

    for (int i = 0; i < expression.length(); i++) {
        char currChar = expression.charAt(i);

        switch (currChar) {
        case '+':
            tokens.add(new Token<>(Type.ADD, String.valueOf(currChar)));
            i++;
            break;

        case '*':
            tokens.add(new Token<>(Type.MUL, String.valueOf(currChar)));
            i++;
            break;

        default:
            if (Character.isWhitespace(currChar)) {
                i++;
            } else {
                String operand = getOperand(expression, i);
                i += operand.length();
                tokens.add(new Token<>(Type.OPERAND, operand));
            }
            break;
        }
    }
    return tokens;
}

public static void main(String[] args) {

    Scanner scan = new Scanner(System.in);
    System.out.println("Please enter an expression: ");
    String input = scan.nextLine();

    ArrayList<Token<Type, String>> tokens = lex(input);
    for (Token token : tokens) {
        System.out.println(token);
    }
}

example of input : "5+1" output :

Token value: 5
Token type: OPERAND
Token value: 1
Token type: OPERAND
4
  • What is your input? What is your expected output? What is the observed output? In general, if you plan to analyze more complex expressions, you might want to design your algorithm in a recursive way.
    – Turing85
    May 17, 2016 at 19:31
  • I edited the input and output. At least I expected the "+" to be recognized as a ADD token, but it's not the case. The further step is to perform the addition, but I did not manage yet.
    – loukios
    May 17, 2016 at 21:56
  • This is not a lexical analysis problem alone. You need a parser as well as a lexical analyser. Look up 'recursive descent expression parser', or the Dijkstra Shunting-yard algorithm.
    – user207421
    May 17, 2016 at 21:57
  • Thanks for your insight, it goes a bit beyond my knowledge for now, but I'll definitely check it.
    – loukios
    May 17, 2016 at 22:22

1 Answer 1

0

In the lex() method, you are incrementing the character position an extra time for each token. You increment after adding the token, but there is an increment in the for loop too.

For example, if your input is "5+1", "5" is added as an OPERAND, the position is updated accordingly, then the for loop skips "+", and the next character is "1", which is added as another operand.

2
  • Thanks! I overlooked that. I notice now that the last token, "1" in the exemple is not taken into account, can you spot why?
    – loukios
    May 17, 2016 at 22:21
  • @loukios No. I took the code in your question currently, changed for (int i = 0; i < expression.length(); i++) { to for (int i = 0; i < expression.length(); ) {, and called lex("5+1") and got the correct results. You must have made some additional changes that broke something else.
    – erickson
    May 17, 2016 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.