4

I'm trying to match either

(\S+)(=)([fisuo])

or

(\S+)(!)

And then have the results placed in a list (capture groups). All of my attempts result in extra, unwanted captures.

Here's some code:

#!/usr/bin/perl
#-*- cperl -*-
# $Id: test7,v 1.1 2023/04/10 02:57:12 bennett Exp bennett $
#

use strict;
use warnings;
use Data::Dumper;

foreach my $k ('debugFlags=s', 'verbose!') {
    my @v;

    # Below is the offensive looking code.  I was hoping for a regex
    # which would behave like this:

    if(@v = $k =~ m/^(\S+)(=)([fisuo])$/) {
      printf STDERR ("clownMatch = '$k' => %s\n\n", Dumper(\@v));
    } elsif(@v = $k =~ m/^(\S+)(!)$/) {
      printf STDERR ("clownMatch = '$k' => %s\n\n", Dumper(\@v));
    }

    @v = ();

    # This is one of my failed, aspirational matches.  I think I know
    # WHY it fails, but I don't know how to fix it.
    
    if(@v = $k =~ m/^(?:(\S+)(=)([fisuo]))|(?:(\S+)(!))$/) {
      printf STDERR ("hopefulMatch = '$k' => %s\n\n", Dumper(\@v));
    }
    printf STDERR "===\n";
}

exit(0);
__END__

Output:

clownMatch = 'debugFlags=s' => $VAR1 = [
          'debugFlags',
          '=',
          's'
        ];


hopefulMatch = 'debugFlags=s' => $VAR1 = [
          'debugFlags',
          '=',
          's',
          undef,
          undef
        ];


===
clownMatch = 'verbose!' => $VAR1 = [
          'verbose',
          '!'
        ];


hopefulMatch = 'verbose!' => $VAR1 = [
          undef,
          undef,
          undef,
          'verbose',
          '!'
        ];


===

There are more details in the code comments. The output is at the bottom of the code section. And the '!' character is just that. I'm not confusing it with some other not.

Update Mon Apr 10 23:15:40 PDT 2023:

With the wise input of several readers, it seems that this question decomposes into a few smaller questions.

Can a regex return a variable number of capture groups?

I haven't heard one way or the other.

Should one use a regex in this way, if it could?

Not without a compelling reason.

For my purposes, should I use a regex to create what is really a lexical-analyzer/parser?

No. I was using a regex for syntax checking and got carried away.

I learned a good deal, though. I hope moderators see fit to keep this post as a cautionary tale.

Everyone deserves points on this one, and can claim that they were robbed, citing this paragraph. @Schwern gets the points for being first. Thanks.

14
  • 1
    One has three captures, one has two captures. How should they be combined?
    – Schwern
    Apr 10 at 3:38
  • I was hoping for @v to be either length 2 or 3 depending on which sub-regex matched, Like the clownMatch examples in the output. Apr 10 at 3:41
  • 1
    Since you're matching two different things, it seems perfectly reasonable to have two different matches. Why do you want to combine them?
    – Schwern
    Apr 10 at 3:42
  • 2
    Using a branch reset w/o undef. try e.g. ^(\S+)(?|(=)([fisuo])|(!)())$ Apr 10 at 8:09
  • 1
    "Using a branch reset w/o undef" -- I don't see how that improves the matter; there is still one extra capture
    – zdim
    Apr 10 at 17:47

4 Answers 4

4

In an alternation the values for all captures are returned, even for those that weren't matched.

An easy way out is to filter out undef's from the return list

if ( my @v = grep { defined } $s =~ /^(?: (\S+)(=)([fisuo]) | (\S+)(!) )$/x )

There are other ways to build the regex as well but a straight-up alternation is just fine.


The question specifically asks how to conflate two (alternative) regex patterns into one in such a way so to get captures only for what is actually matched, without extra undef's. This is a good question in my opinion as it would often be nice to not have to clean up.

The usual alternation (p1 | p2) returns (in a list context or in @{^CAPTURE}) all indicated capture groups, as stated above. If p1 defines three capture groups and p2 two, in the end we get five; captures for the branch that matched and undefs for the other.

In short, I find that to get a "clean" set of true captures only, with a pure-regex, we need to parse with a grammar. While the builtin support (see DEFINE) can only match ("recognize") patterns, the Regexp::Grammars supports far more. A simple example is suitable

use warnings;
use strict;
use feature 'say';
use Data::Dump qw(dd);  # Data::Dumper is in the core

my $grammar = do {
    use Regexp::Grammars;
    qr{ 
        <word> <symb> <val>?

        <nocontext:>
        <token: word>  [^=!]+  # or use \w in a character class with chars
                               # that are also allowed, like [\w-.] etc

        <token: symb>  = | !
        <token: val>   [fisuo]
    }x;
};

for my $s (qw(debugFlags=s verb!)) {
    if ($s =~ $grammar) { 
        dd \%/;              # hash %/ is populated with results
        say for values %/;   # just the results
        say '-'x60;
    }   
}

This prints

{ symb => "=", val => "s", word => "debugFlags" }
s
=
debugFlags
------------------------------------------------------------
{ symb => "!", word => "verb" }
!
verb
------------------------------------------------------------

The results aren't sorted so one may want to add a desired sorting criterion for the hash, or go through the individual hash elements.

The example in the question is very simple so a trivial grammar works for it, but if we imagine it growing to process options more comprehensively then the grammar would need to be more involved/structured. For example, while this is still simple

qr{
    <option>   # run the matching

    # Define the grammar
    <nocontext:>
    <token: option>     <opt_vals> | <opt_flag>

    <token: opt_vals>   <word> <symb_vals> <val>
    <token: opt_flag>   <word> <symb_flag>?

    <token: word>       [^=!:]+

    <token: symb_vals>  = | :
    <token: symb_flag>  !
    <token: val>        [fisuo]

}x;

it can be expanded more easily and it is more precise.

The aim of regex in this question is to check usage of Getopt::Long, a module for parsing command-line options, and there can be nothing following ! (negation for flag-type options). So symbols following names of options with values (= and :) are separated from !. There is of course a lot more in the library's syntax; this is a demo.

Please see the (seemingly endless) docs for the many, many Regexp::Grammars features, of which practically none are used here.


All else seems to suffer from extra undefs. The "branch reset" comes close but still returns the longest set of indicated capture groups (3 here) even when it matches a shorter branch, as I mentioned in the comment below; so we get undefs again. See the answer from @LanX for how to use this.

The conditional expression, for which I hoped that it might dodge the bullet, also sets up all capturing parentheses that it sees

for (qw(debugFlags=s verb!)) 
{
    if ( /^([^=!]+) (?(?==) (=)([fisuo]) | (!))$/x ) {
        say "Length: ", scalar @{^CAPTURE};
        say $_//'undef' for @{^CAPTURE};
    }
}

We duly get two undef printed in the second test. I use a lookahead for the condition specifically to try to avoid extra capture groups but of course all parens further in the expression get them, regardless of which ones match. So one can really do

    if ( /^([^=!]+) (=)? (?(2) ([fisuo]) | (!))$/x )

(with same results, good match and capture but with extra undefs)

7
  • That's one way to do it. I was hoping to learn more about the regex way. I upvoted this anyway, though, because I hadn't used grep in that way before, and I love to learn new things. Apr 10 at 3:57
  • 2
    @ErikBennett "the regex way" -- I can't think of a generic and straightforward way which won't return all introduced captured groups, even those that are undef as they didn't match (in another branch). The branch reset pattern comes close but still returns the longer capture group (3 here), even when it matches the shorter branch. (So if it matches verbose! it still returns three-long list of captures, one being undef). Then there are many ways to craft a regex to avoid alternation but then that depends on the particular pattern.
    – zdim
    Apr 10 at 4:19
  • This is going to take me some time to study. From the looks of it, it may catch foo!s, but I really need to read up on this all. I'll keep watching and asking. This is turning out to be a bigger deal that the one liner I was expecting. I love it. Apr 11 at 20:01
  • It might catch bar=, as well. But this should just be a matter of simple changes. Dang, I haven't needed this stuff since school. At the risk of dating myself, that was before the "Camel Book". Apr 11 at 20:16
  • @ErikBennett The actual regex depends on the exact use case and is a matter of choice, yes. With \S+ you catch everything and rely on backtracking with the what follows (= etc). With \w+ it's missing - for sure, with [\w-]+ perhaps some other character which can be expected in an option name. A nice choice is to list what cannot be there, which is what I did -- but then that probably needs tweaking. (What other symbols can be used after the name? There is : for instance, so that should be added, for [^=!:]+. Etc) But that's the easy part, to craft the exact subpattern
    – zdim
    Apr 11 at 23:18
3

We can use the following single regex pattern:

^(\S+)([!=])((?<==)[fisuo])?$

This says to match:

  • ^ from the start of the string
  • (\S+) match and capture in $1 a non whitespace term
  • ([!=]) match and capture in $2 either ! or =
  • ((?<==)[fisuo])? then capture optionally in $3 a letter from fisuo the lookbehind (?<==) ensures this only matches for =
  • $ end of the string

Demo

3
  • Lookbehind! I was messing with lookaheads, but didn't want to publish my failed attempts. This still leaves a trailing undef on the 2 group ex. (verbose!), but it's certainly going to work. For yuks, is there anyway to have it (or any regex) return 2 or 3 (or variable number) groups? Apr 10 at 3:54
  • @ErikBennett If you want an empty string instead of undef with this, an idea to make the pattern inside optional e.g. by removing the ? and adding OR nothing |) at the end. Apr 10 at 9:00
  • 1
    @bobblebubble "If you want an empty string instead of undef..." -- but that still doesn't give what the question asks, a list of actual captures from the branch that matched (either 2 or 3 in this case). Not undef's or bogus empty strings. This just doesn't answer that.
    – zdim
    Apr 10 at 9:18
3

All of my attempts result in extra, unwanted captures.

I'd go for the "branch reset" (?| pattern1 | pattern2 | ... ) like already suggested by @bobble_bubble (as comment only)

It's a generic solution to combine different patterns with groups, while resetting the capture-count.

Alas contrary to the docs he linked to, you'll still get undef slots at the end of the LISTs returned for patterns with less groups.

But if this really bothers you - personally I would keep them - you can safely filter them out with a grep {defined} like @zdim suggested.

That's safe since undef means a non-match and can't be confused with an empty match "".

Here the code covering your test cases

use v5.12.0;
use warnings;
use Data::Dump qw/pp ddx/;
use Test::More;

# https://stackoverflow.com/questions/75974097/merge-two-regexes-with-variable-number-of-capture-groups

my %wanted =
  (
   "debugFlags=s" => ["debugFlags", "=", "s"],
   "verbose!"     => ["verbose", "!"],
  );


while ( my ( $str, $expect) = each %wanted ) {
    my @got =
      $str =~ / (\S+)
                (?|
                    (=) ([fisuo]+)
                |
                    (!)
                )
              /x;

    ddx \@got;                          # with trailing undefs

    @got = grep {defined} @got;         # eliminate undefs

    is_deeply( \@got, $expect, "$str => ". pp(\@got));
}

done_testing();

-->

# branchreset.pl:25: ["debugFlags", "=", "s"]
ok 1 - debugFlags=s => ["debugFlags", "=", "s"]
# branchreset.pl:25: ["verbose", "!", undef]
ok 2 - verbose! => ["verbose", "!"]
1..2

strategic update

But again, I don't see the point in eliminating the undef slots at the end, since you will need to handle the different cases individually anyway.

And one day you might want to add patterns after the branch too. If branch-reset was really skipping the missing groups, that would change the numbering of trailing groups beyond recognition. So from a design perspective that's well done.

11
  • "I'd go for the "branch reset" (?| pattern1 | pattern2 | ... ) like already suggested by @bobble_bubble (as comment only)" -- But, as I stated when I mentioned this in a comment under my answer, it still returns the list of the length of the longest branch, so it decidedly returns undef values as well (when the shorter branch matches). This does not answer the question. Btw, this is not contrary to the docs I linked to.
    – zdim
    Apr 10 at 17:43
  • 1
    @zdim I addressed this at length, please read the whole answer. And I didn't refer to the docs you linked to.
    – LanX
    Apr 10 at 18:01
  • "please read the whole answer" -- alright, did so carefully this time (and I did miss a few bits). Still, I am not sure what this answer aims for: the question asks for how to conflate patterns with different numbers of captures, so that you get back actual captures, no undefs. That's precisely the question, and it's a good one; it'd be nice to avoid those 'undef`s. This answer is a nice discussion, and with some extras, but it doesn't answer the question. From what I see this simply offers a different approach, perhaps with some benefits, but which suffers the same problem.
    – zdim
    Apr 10 at 19:26
  • Then you "don't see the point in eliminating undef"... well, maybe that is indeed misplaced, but that was the question. Some other answers here seem to just ignore that, and I'll repeat my opinion: it is a good question. I don't see why it's considered irrelevant -- getting those undef is a nuisance for which we normally have to do something. It'd be nice to not have to. (As for their comment, that came in reference to my original comment, they indeed link to different docs. My bad. I recommend referring to perldoc, like you do.)
    – zdim
    Apr 10 at 19:29
  • 1
    3. Branch-reset is worth to have a prominent answer and not being hidden in some comments. Other searching here might just want that.
    – LanX
    Apr 10 at 19:38
2

Since you're matching two different things, it seems perfectly reasonable to have two different matches.

But, if you do want to combine them, you can do this:

m{^
  (\S+)
  (?:
    =([fisuo]) |
    (!)
  )
  $
}x

$1 is the name. $2 is the switch, if present. $3 is the !, if present.

For anything more complicated, use named captures or Regexp::Assemble.

Demonstration

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.