4

Say I have the following strings (inputs) in python:

1) "$ 1,350,000" 2) "1.35 MM $" 3) "$ 1.35 M" 4) 1350000 (now it is a numeric value)

Obviously the number is the same although the string representation is different. How can I achieve a string matching or in other words classify them as equal strings?

One way would be to model -using regular expressions- the possible patterns. However there might be a case that I haven't thought of.

Does someone see a NLP solution to this problem?

Thanks

6
  • 1
    I'm not sure it's an NLP problem but I've never seen "MM" for million.
    – roganjosh
    Jan 20, 2018 at 17:55
  • @abra interesting. Is that in use in America? Other than that, which I'm not familiar with, it should be possible to generate some rules to get at these numbers. Maybe there's another "MM" type I'm not familiar with...
    – roganjosh
    Jan 20, 2018 at 17:59
  • This is not an NLP problem, just a job for regexes, plus some code to ignore order, and lookup a dictionary of known abbreviations(/ontology) like "MM". I don't know what you expect an NLP solution would look like, unless we check the word(s) after the number to see if they're synonyms for "million", as "MM" is (in finance). i.e. we lookup an ontology. Why do people abbreviate "million" as "mm"?
    – smci
    Jan 20, 2018 at 18:12
  • By the way, we can completely disregard the '$' character here (unless you need to disambiguate against other currencies or symbols). So all this boils down to is parsing number formats, and mapping 'MM'/'million' -> multiply by 1e6. And doing that parsing in an order-independent way.
    – smci
    Jan 20, 2018 at 18:19
  • Thanks for the comments so far. Yes indeed regex seem to do the work. I haven't worked on NLPs before but the problem i am solving has to do with text mining from a very long document that is practically not feasible to parse it manually. That's why I asked
    – mrt
    Jan 20, 2018 at 18:21

3 Answers 3

5

This is not an NLP problem, just a job for regexes, plus some code to ignore order, and lookup a dictionary of known abbreviations(/ontology) like "MM".

  • First, you can completely disregard the '$' character here (unless you need to disambiguate against other currencies or symbols).
  • So all this boils down to is parsing number formats, and mapping 'M'/'MM'/'million' -> a 1e6 multiplier. And doing that parsing in an order-independent way (e.g. the multiplier, currency symbol and amount can appear in any relative order, or not at all)

Here's some working code:

def parse_numeric_string(s):

    if isinstance(s, int): s = str(s)

    amount = None
    currency = ''
    multiplier = 1.0

    for token in s.split(' '):

        token = token.lower()

        if token in ['$','€','£','¥']:
            currency = token

        # Extract multipliers from their string names/abbrevs
        if token in ['million','m','mm']:
            multiplier = 1e6
        # ... or you could use a dict:
        # multiplier = {'million': 1e6, 'm': 1e6...}.get(token, 1.0)

        # Assume anything else is some string format of number/int/float/scientific
        try:
            token = token.replace(',', '')
            amount = float(token)
        except:
            pass # Process your parse failures...

    # Return a tuple, or whatever you prefer
    return (currency, amount * multiplier)

parse_numeric_string("$ 1,350,000")
parse_numeric_string("1.35 MM $")
parse_numeric_string("$ 1.35 M")
parse_numeric_string(1350000)
  • For internationalization, you may want to beware that , and . as thousands separator and decimal point can be switched, or ' as (Arabic) thousands separator. There's also a third-party Python package 'parse', e.g. parse.parse('{fn}', '1,350,000') (it's the reverse of format())
  • Using an ontology or general NLP library would probably be way more trouble than it's worth. For example, you'd need to disambiguate between 'mm' as in "accounting abbreviation for millions" vs "millimeters" vs 'Mm' as in 'Megameters, 10^6 meters' which is an almost-never-used but valid metric unit for distance. So, less generality probably better for this task.
  • and you could also use a dict-based approach to map other currency signifiers e.g. 'dollars','US','USD','US$', 'EU'...
  • here I tokenized on whitespace, but you might want to tokenize on any word/numeric/whitespace/punctuation boundaries so you can parse e.g. USD1.3m
2
  • I still find M much more common and standard as an abbreviation compared to MM
    – Cadoiz
    Jan 6, 2022 at 16:24
  • @Cadoiz: it depends on what you're reading: news articles, company accounts...
    – smci
    Jan 6, 2022 at 23:09
1

Interesting question and here's my solution.
You could write a little class which looks for potential matches, separates them in an amount and a unit and tries to convert them afterwards:

import re, locale, math

# us
locale.setlocale(locale.LC_ALL, 'en_US')
from locale import atof

data = """
Say I have the following strings (inputs) in python:

1) "$ 1,350,000" 2) "1.35 MM $" 3) "$ 1.35 M" 4) 1350000 (now it is a numeric value)

Obviously the number is the same although the string representation is different. How can I achieve a string matching or in other words classify them as equal strings?

One way would be to model -using regular expressions- the possible patterns. However there might be a case that I haven't thought of.

Does someone see a NLP solution to this problem?
Thanks

here might be some other digits: 1.234
"""

class DigitMiner:
    def __init__(self):
        self.numbers = []

    def convert(self, amount, unit):
        if unit in ['M', 'MM']:
            amount *= 10**6
        elif unit in ['K']:
            amount *= 10**3
        else:
            pass
        return amount

    def search(self, string=None):
        rx = re.compile(r'''
            (?P<amount>\b\d[\d.,]+\b)\s*
            (?P<unit>M*)''', re.VERBOSE)

        for match in rx.finditer(string):
            amount = self.convert(atof(match.group('amount')), match.group('unit'))
            if amount not in self.numbers:
                self.numbers.append(amount)


dm = DigitMiner()
dm.search(data)
print(dm.numbers)

This yields:

[1350000.0, 1.234]


Note that locale.atof() converts a string to a floating point number, following the LC_NUMERIC settings.

0

Consider creating a routine which matches the string input (which can be in any 4 of the given formats) against 4 regex patterns like so:

For "$ 1,350,000":

(?<=\$ )([\d,]+)

For "1.35 MM $":

([\d\.]+)(?= MM \$)

For "$ 1.35 M":

(?<=\$ )([\d\.]+)(?= M)

For 1350000:

([\d]+)

And then convert these matches into ints to be returned and compared with others.

The regex patterns provided will match only the digits commas and decimal places of the strings (by using lookaheads & lookbehinds).

Note: depending on which regex gets a match, will require the extracted digits to be processed accordingly. (e.g. 1.35 from "$ 1.35 M" needs to be multiplied by 1000000 before being returned)

Not the answer you're looking for? Browse other questions tagged or ask your own question.