Questions tagged [lexical-analysis]
Process of converting a sequence of characters into a sequence of tokens.
798
questions
0
votes
0
answers
17
views
Unexpected character JFLEX
Hello i was watching a tutorial on youtube. It was the start of tutorial but when i tried to wrote the code and run this flex.l file. The output show unexpected character [a-z] {printf("Single ...
-1
votes
1
answer
25
views
Regular expression for email in flex
I am trying to wirte a regular expression for emails in JFlex. So far I tried with this
L=[a-zA-Z_]+
D=[0-9]+
email=[^(.+)@(\S+)$]
%{
public String lexeme;
%}
%%
{L}({L}|{D})* {lexeme=yytext();...
1
vote
1
answer
24
views
What is a valid character in an identifier called?
Identifiers typically consist of underscores, digits; and uppercase and lowercase characters where the first character is not a digit. When writing lexers, it is common to have helper functions such ...
0
votes
1
answer
31
views
Jflex ambiguity
I have these two rules from a jflex code:
Bool = true
Ident = [:letter:][:letterdigit:]*
if I try for example to analyse the word "trueStat", it gets recognnized as an Ident expression and ...
0
votes
0
answers
16
views
Optional characters in Flex expression
I'm trying to make a flex expression which identifies money, to 2 decimal places. This isn't for any practical use, I'm just messing around with flex and trying to learn it.
I am looking for money in ...
-1
votes
1
answer
59
views
Regex in c++ for maching some patters
I want regex of this.
add x2, x1, x0 is a valid instruction;
I want to implement this. But bit confused, how to, as I am newbie in using Regex. Can anyone share these Regex?
1
vote
1
answer
35
views
Using flex to identify variable name without repeating characters
I'm not fully sure how to word my question, so sorry for the rough title.
I am trying to create a pattern that can identify variable names with the following restraints:
Must begin with a letter
...
0
votes
1
answer
130
views
How To Count All Operands And Operators in a Java File
I have a java file and i want to count the number of binary,unary,logical operators,and operands in it.I just know i need to use regex for it and i know the basic of it.The task might be simple but ...
2
votes
2
answers
64
views
Evaluation order of augmented operators (delimiters) in python
If I evaluate the following minimal example in python
a = [1, 2, 3]
a[-1] += a.pop()
I get
[1, 6]
So it seems that this is evaluated as
a[-1] = a[-1] + a.pop()
where each expression/operand would ...
-2
votes
1
answer
171
views
How do I find the regular expression for- All binary numbers greater than 110011
I am trying to solve a problem set to practice for an exam. How can I approach questions like these ? Is there a way to verify solutions or is it just trial and error ?
0
votes
1
answer
239
views
Lex program to count number of strings having ‘a’ as the second character
I wrote a program to count the number of strings having 'a' as the second character of the string. I tried using several patterns of the lex operators and also tried using the lex program start ...
0
votes
1
answer
90
views
flex filename.l won't produce lex.yy.c
I want to detect if there are any duplicates of uppercase letters from input. The program should work fine but when i try to run the command flex filename.l on cmd windows it won't show any errors or ...
0
votes
0
answers
42
views
Is there a version of Java that supports lexically extracting the name of a local variable, so that it is available at runtime?
Is there a version of Java that supports a syntax similar to this?:
Integer myInteger = new Integer(0);
if (debug) {
System.out.println($myInteger + ": " + myInteger);
}
where
$...
0
votes
0
answers
22
views
How called quotes of string litherals in lexical analysis?
In C-like languages, the string litherals are the arbitrary characters surrouned with quotes (e. g. "example"). But what is "? I know that it is terminal, or therminal character, but is ...
2
votes
1
answer
67
views
Lex file doesn't recognize keywords. How can I write my regex to read particular keywords?
This is my lex file
%{
#include<stdio.h>
int codelines = 0;
int commentlines = 0;
int blanklines = 0;
int headerlines = 0;
int brackets = 0;
int keywords = 0;
%...
-1
votes
1
answer
33
views
based word for battling and lemmatization
All,
What is the base form of battling? Lemmatization results in battling where as I think it should be battle. Is my understanding of lemmatization wrong?
from nltk import download
download('wordnet')...
-1
votes
1
answer
137
views
Flex C++ print code in generated file and add color
I'm trying to insert the code that flex reads into my .tex file, this console app is supposed to take a .pascal and analyze it and then generate a .tex file but I'm not able to pass the code to the ....
0
votes
1
answer
41
views
Make categories on my qualitative variable column
I need to categorise my qualitative variables into groups:
For example: on one of my qualitative columns ("packaging"),
I have to group all my plastic data like "Plastic", "...
0
votes
0
answers
23
views
reading python indentations in files within a project
In the Python documentation here are explained how indentations are computed during coding with Python. Here is another explanation, plus this:
For example:
if foo:
if bar:
x = 42
else:
...
0
votes
1
answer
191
views
How can i ignore comments in a string based on compiler design?
I want to ignore every comment like { comments } and // comments.
I have a pointer named peek that checks my string character by character. I know how to ignore newlines, tabs, and spaces but I don'...
1
vote
0
answers
38
views
Creating grammars for lexical analyzer
I am trying to create a lexical analyzer for a given language. Can't write grammar rules correctly. My task is
The input language contains conditional statements if ... then ... else and if ... then, ...
0
votes
0
answers
40
views
How to determine the goal symbol when tokenize ECMAScript in lexical grammar
I'm writing a JS macro preprocessor, and refer to the ECMA-262 specification for some guide to parse js code.
I use the code parser to determine where to do macro replace in a standard/precise way, ...
-1
votes
1
answer
116
views
What does [^0-9]+$ mean (regular expression in FLEX)
This is what I know:
^ inside brackets matches a character that isn't one of the included inside the brackets.
+ Matches one or more appearances of the expression to its left (in my ex. [^0-9]).
$ If ...
0
votes
1
answer
94
views
Change input of flex from stdin to file?
So I've been using flex for awhile, and the entire time I have been feeding it input directly from the terminal. Now I want to feed it a file. An adhoc solution I came up with was to use dup to change ...
-1
votes
2
answers
104
views
antlr 4 lexer rule RULE: '<TAG>'; isn't recognized as token but if fragment rule then recognized
EDIT:
I've been asked if I can provide the full grammar. I cannot and here is the reason why:
I cannot provide my full grammar code because it is homework and I am not allowed to disclose my solution, ...
0
votes
2
answers
89
views
Patterns for making two packages as loosely coupled as possible
So I'm learning to build a parser for a intro class for software design.
I have built a tokenizer package that splits my input string into Token objects that have getters for both the value and type. ...
0
votes
2
answers
113
views
Characters and digits of Chapter four of the Unicode Standard
In a language specification, there is
name-start-character=
'_' | '\' | ? any code points which are characters as defined by the Unicode character properties, chapter four of the Unicode Standard ?;
...
3
votes
1
answer
173
views
Check if a sequence is composed of two identical sequences
The problem is to find for a 1<=n<=50000-element sequence of numbers (1 to 10^5) if such a subsequence can be chosen that the remaining elements will create another copy of the chosen ...
0
votes
0
answers
92
views
Could someone explain the NFA to DFA transform in flex-lexer?
I have a very simple test.l file shown as below:
...
%%
[aeiou]+ {printf("aeiou");}
^r {printf("^r");}
%%
...
I used flex-2.6.0 to create the lex.yy.c ...
1
vote
1
answer
445
views
Parser gives a unknown type name error for entry_t
I'm editing my first parser and I'm very new to compiler design. I'm using a hashtable to store tokens. I have created a struct called struct_t in my symboletable.h file.
When I try to create a new ...
0
votes
1
answer
175
views
Lexical analyzer gives undefined error for yylineno
Im pretty new to compiler design. im trying to take the first step in lexical analyzer trying to run the following code.
%{
#include <stdlib.h>
#include <stdio.h>
#include "...
2
votes
1
answer
140
views
Skipping nested comments recursively in a scanner
I'm writing a scanner for a compiler and have a function to skip comments whenever it sees them, I wanted to know how one would skip nested comments, i.e something like " hello "w" ...
1
vote
1
answer
102
views
How can I get all modulo(%) operation in my C source code
How can I get all modulo(%) operation in my c source code?
I don't think a regular expression can do it, due to C MACRO inclusion.
And, must take care of string printf formatting.
Which static analyse ...
1
vote
1
answer
69
views
Tokenizing f string in lexical analysis phase of Python language
I have recently learned about compilers through Aho's dragon book. As I understand it languages are first put through lexical analysis for tokenization. This process seems simple enough for languages ...
0
votes
1
answer
53
views
Why is the assignment operator = not listed under operators in the Python official doc on lexical analysis?
I was reading through the Python lexical specification from the offical docs. I was surprised to discover that the assignment operator = is not listed under the operators' section. Many other ...
-1
votes
1
answer
63
views
Extract variable names from java expression dinamically
I have dynamic set of string expressions(java code parts)
eg :
val1.subtract(val2.divide(val3,6,java.math.RoundingMode.HALF_UP) ,new MathContext(6, java.math.RoundingMode.HALF_UP)).setScale(6, ...
1
vote
0
answers
159
views
Removing heading, leading, and repeated whitespace but do not remove whitespace in lex and bison
I am currently working on a preprocessor of some sort and am using lex and bison to create the lexer and parser part.
And my current implementation has a problem handling whitespace. My first solution ...
-1
votes
1
answer
376
views
How to develop a lexical analyzer with javascript? [closed]
I developed a lexical analyzer function which gets a string and separate the items in string in an array like this :
const lexer = (str) =>
str
.split(" ")
.map((s) => s.trim(...
0
votes
1
answer
94
views
How to change "start condition" outside of actions in flex/lex?
As the title states, how do I change the currently active start condition or state in (f)lex outside of action code? Flex manual states that special directives like BEGIN can only be used inside an ...
0
votes
0
answers
99
views
Scala Parser Combinator lexer handling for unterminated string
I am wondering how to handle unterminated strings in a lexer/scanner written using the Scala Parser Combinator library. The example in class scala.util.parsing.combinator.lexical.StdLexical doesn't ...
1
vote
1
answer
138
views
Why we count a string as a single token in lexical analysis of compiler design?
I am learning about compiler design. The task of lexical analyser in compiler is to convert the code into stream of tokes. But I am confused why we consider a string as a single token . For example - ...
0
votes
1
answer
62
views
How to take multiple output file name as a command line argument in lexical analyzer FLEX?
I want to take three file name as a command line argument for lexical analyzer FLEX , among these three one for input file and remaining two for output file. It is easy to take one input file name and ...
0
votes
2
answers
217
views
What is the regular expression for matching white space among words in a sentence for FLEX?
I want to first match the white spaces among words then want to count total number of white spaces in a sentence. I am trying in this below way ,
%{
#undef yywrap
#define yywrap() 1
int VowelCount = 0;...
0
votes
1
answer
73
views
Getting Warning: rule cannot be matched
So i am trying to make e lexical parsher using Flex Code with a bit of Bison.I keep getting a warning about my rules than cannot be matched and i cannot find any error.I have also searched what that ...
3
votes
1
answer
102
views
How to get the list of negative words of certain verbs for Yes/No Questions?
I have a list of verbs:
words_list =['do','does','did','was'...............]
I want to get a new list of verbs including all the negative of the above words. For example:
negative_words = ["don'...
0
votes
2
answers
66
views
regex detecting single line
Need regex for detecting string in only single line,i tried this regex \"(\\.|[^"\\])*\" but this works also with multiline
input:
Example :
this is a "string
and the string ends ...
2
votes
0
answers
117
views
How to store and use data from a text file in python for sentiment-analysis?
So I'm writing a sentimental analysis program where I would be summing up the emotions of a character in a text file based on the words used in the sentence. I would be using the data from NRC Emotion ...
0
votes
0
answers
84
views
how to tokenize input based on a set of ABNF grammar rules
I have read the RFC on the ABNF specification and am having difficulty understanding how a set of ABNF rules could be used to reliably extract tokens from some input string that matches the grammar. ...
1
vote
0
answers
38
views
How do deal with the "type context" rule (Java SE 16 spec §3.2) for `>` characters properly when performing lexical analysis?
I'm recently interested in writing a parser for Java language. My reference is newly released Java SE 16 spec. On 3.2. Lexical Translations, I come across this sentence but I'm not sure whether I get ...
1
vote
0
answers
41
views
How do i remove a nextline symbol # from its output in java?
I've been creating a lexical analyzer and everything works fine, except, I don't want the output to include a nextline "#" in the console. How can I do this ?
Token getToken() {
int ...