5

I'm trying to write a text normalizer, and one of the basic cases that needs to be handled is turning something like 3.14 to three point one four or three point fourteen.

I'm currently using the pattern \$?\d+(\.\d+)?%? with nltk.regexp_tokenize, which I believe should handle numbers as well as currency and percentages. However, at the moment, something like $23.50 is handled perfectly (it parses to ['$23.50']), but 3.14 is parsing to ['3', '14'] - the decimal point is being dropped.

I've tried adding a pattern separate \d+.\d+ to my regexp, but that didn't help (and shouldn't my current pattern match that already?)

Edit 2: I also just discovered that the % part doesn't seem to be working correctly either - 20% returns just ['20']. I feel like there must be something wrong with my regexp, but I've tested it in Pythex and it seems fine?

Edit: Here is my code.

import nltk
import re

pattern = r'''(?x)    # set flag to allow verbose regexps
            ([A-Z]\.)+        # abbreviations, e.g. U.S.A.
            | \w+([-']\w+)*        # words w/ optional internal hyphens/apostrophe
            | \$?\d+(\.\d+)?%?  # numbers, incl. currency and percentages
            | [+/\-@&*]         # special characters with meanings
            '''
    words = nltk.regexp_tokenize(line, pattern)
    words = [string.lower(w) for w in words]
    print words

Here are some of my test strings:

32188
2598473
26 letters from A to Z
3.14 is pi.                         <-- ['3', '14', 'is', 'pi']
My weight is about 68 kg, +/- 10 grams.
Good muffins cost $3.88 in New York <-- ['good', 'muffins', 'cost', '$3.88', 'in', 'new', 'york']
3
  • ? is non greedy (miminum match) try replacing it with {,1}
    – vish
    Mar 4, 2014 at 15:26
  • Try this: (\$?\d+(?:\.\d+)?%?) Mar 4, 2014 at 15:36
  • Could you provide the code you use for testing this? I can't reproduce that issue here, both $23.50 and 3.14 are handled fine
    – gonz
    Mar 4, 2014 at 15:50

2 Answers 2

7

The culprit is:

\w+([-']\w+)*

\w+ will match numbers and since there's no . there, it will match only 3 in 3.14. Move the options around a bit so that \$?\d+(\.\d+)?%? is before the above regex part (so that the match is attempted first on the number format):

(?x)([A-Z]\.)+|\$?\d+(\.\d+)?%?|\w+([-']\w+)*|[+/\-@&*]

regex101 demo

Or in expanded form:

pattern = r'''(?x)               # set flag to allow verbose regexps
              ([A-Z]\.)+         # abbreviations, e.g. U.S.A.
              | \$?\d+(\.\d+)?%? # numbers, incl. currency and percentages
              | \w+([-']\w+)*    # words w/ optional internal hyphens/apostrophe
              | [+/\-@&*]        # special characters with meanings
            '''
5
  • GREAT catch! It's easy to forget that order matters in matching. Mar 4, 2014 at 17:35
  • This is it. Great answer. Yes order matters, in this king of simple ORs once the regex start matching it'll keep consuming your string, it's not the most specific match it's the first that matches the current character.
    – gonz
    Mar 4, 2014 at 17:38
  • @JessicaYang Well, if you didn't put up the code, I wouldn't have ever guessed :) You're welcome! ^^
    – Jerry
    Mar 4, 2014 at 17:40
  • @Jerry Making that change broke my ordinal number ('1st', '322nd') parser - ['1','st'] - and tacking a \b or $ onto the end of it brings my ordinals back, but now percentages don't work like I'd expect (20% is captured only as 20). Do you know what might be causing that? Mar 4, 2014 at 17:59
  • 2
    @JessicaYang \b doesn't match between % and a space because they're both non-word characters. Try using this for the number part: \$?\d+\b(\.\d+)?%? It will solve the current issues you have, but I don't know if it'll solve future ones. Maybe you need a different approach as well =/
    – Jerry
    Mar 4, 2014 at 17:59
0

Try this regex:

\b\$?\d+(\.\d+)?%?\b

I surround the initial regex with word boundaries matching: \b.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.