2

Hello all I'm trying to parse out a pretty well formed string into it's component pieces. The string is very JSON like but it's not JSON strictly speaking. They're formed like so:

createdAt=Fri Aug 24 09:48:51 EDT 2012, id=238996293417062401, text='Test Test', source="Region", entities=[foo, bar], user={name=test, locations=[loc1,loc2], locations={comp1, comp2}}

With output just as chunks of text nothing special has to be done at this point.

createdAt=Fri Aug 24 09:48:51 EDT 2012 
id=238996293417062401 
text='Test Test' 
source="Region"
entities=[foo, bar] 
user={name=test, locations=[loc1,loc2], locations={comp1, comp2}}

Using the following expression I am able to get most of the fields separated out

,(?=(?:[^\"]*\"[^\"]*\")*(?![^\"]*\"))(?=(?:[^']*'[^']*')*(?![^']*'))

Which will split on all the commas not in quotes of any type, but I can't seem to make the leap to where it splits on commas not in brackets or braces as well.

13
  • 2
    JSON (including this version of it in your question) is not matchable by a regular expression. If your "JSON" is always the same, you may be able to get away with it. Is this always the case?
    – Brian
    Feb 5, 2013 at 18:45
  • No it's not the case... It can differ pretty wildly, but there will never be a comma outside of quotes, brackets, or braces that I don't need to spit on Feb 5, 2013 at 18:49
  • 1
    Can brackets or braces be nested? If so then what you're trying to match is not regular. (While many regex implementations, including Java's, are more powerful than true regular expressions, exploiting this fact is rarely a good idea.) Feb 5, 2013 at 18:52
  • It'd easier if you could show us what would be the output Feb 5, 2013 at 18:56
  • @LaurenceGonsalves Yes unfortunately they can be. Feb 5, 2013 at 19:16

2 Answers 2

2

Because you want to handle nested parens/brackets, the "right" way to handle them is to tokenize them separately, and keep track of your nesting level. So instead of a single regex, you really need multiple regexes for your different token types.

This is Python, but converting to Java shouldn't be too hard.

# just comma
sep_re = re.compile(r',')

# open paren or open bracket
inc_re = re.compile(r'[[(]')

# close paren or close bracket
dec_re = re.compile(r'[)\]]')

# string literal
# (I was lazy with the escaping. Add other escape sequences, or find an
# "official" regex to use.)
chunk_re = re.compile(r'''"(?:[^"\\]|\\")*"|'(?:[^'\\]|\\')*[']''')

# This class could've been just a generator function, but I couldn;'t
# find a way to manage the state in the match function that wasn't
# awkward.
class tokenizer:
  def __init__(self):
    self.pos = 0

  def _match(self, regex, s):
    m = regex.match(s, self.pos)
    if m:
      self.pos += len(m.group(0))
      self.token = m.group(0)
    else:
      self.token = ''
    return self.token

  def tokenize(self, s):
    field = '' # the field we're working on
    depth = 0  # how many parens/brackets deep we are
    while self.pos < len(s):
      if not depth and self._match(sep_re, s):
        # In Java, change the "yields" to append to a List, and you'll
        # have something roughly equivalent (but non-lazy).
        yield field
        field = ''
      else:
        if self._match(inc_re, s):
          depth += 1
        elif self._match(dec_re, s):
          depth -= 1
        elif self._match(chunk_re, s):
          pass
        else:
          # everything else we just consume one character at a time
          self.token = s[self.pos]
          self.pos += 1
        field += self.token
    yield field

Usage:

>>> list(tokenizer().tokenize('foo=(3,(5+7),8),bar="hello,world",baz'))
['foo=(3,(5+7),8)', 'bar="hello,world"', 'baz']

This implementation takes a few shortcuts:

  • The string escapes are really lazy: it only supports \" in double quoted strings and \' in single-quoted strings. This is easy to fix.
  • It only keeps track of nesting level. It does not verify that parens are matched up with parens (rather than brackets). If you care about that you can change depth into some sort of stack and push/pop parens/brackets onto it.
1

Instead of splitting on the comma, you can use the following regular expression to match the chunks that you want.

(?:^| )(.+?)=(\{.+?\}|\[.+?\]|.+?)(?=,|$)

Python:

import re
text = "createdAt=Fri Aug 24 09:48:51 EDT 2012, id=238996293417062401, text='Test Test', source=\"Region\", entities=[foo, bar], user={name=test, locations=[loc1,loc2], locations={comp1, comp2}}"
re.findall(r'(?:^| )(.+?)=(\{.+?\}|\[.+?\]|.+?)(?=,|$)', text)

>> [
    ('createdAt', 'Fri Aug 24 09:48:51 EDT 2012'), 
    ('id', '238996293417062401'), 
    ('text', "'Test Test'"), 
    ('source', '"Region"'), 
    ('entities', '[foo, bar]'), 
    ('user', '{name=test, locations=[loc1,loc2], locations={comp1, comp2}}')
   ]

I've set up grouping so it will separate out the "key" and the "value". It will do the same in Java - See it working in Java here:

http://www.regexplanet.com/cookbook/ahJzfnJlZ2V4cGxhbmV0LWhyZHNyDgsSBlJlY2lwZRj0jzQM/index.html

Regular Expression explained:

  • (?:^| ) Non-capturing group that matches the beginning of a line, or a space
  • (.+?) Matches the "key" before the...
  • = equal sign
  • (\{.+?\}|\[.+?\]|.+?) Matches either a set of {characters}, [characters], or finally just characters
  • (?=,|$) Look ahead that matches either a , or the end of a line.
3
  • 1
    It is going to fail badly if the quoted string contains ,.
    – nhahtdh
    Feb 5, 2013 at 22:38
  • Considering this shouldn't be solved with regular expressions, it's as close as I could get. If you have any suggestions on how I can improve upon this let me know.
    – Jay
    Feb 7, 2013 at 18:33
  • It can be resolved by matching quoted string (quite common on SO): "(?:[^"\\]|\\.)+". But you are right when you said it shouldn't be solved with regular expression - since what I pointed out is not the only case that the regex may fail.
    – nhahtdh
    Feb 7, 2013 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.