1

I'm building a small utility method that parses a line (a string) and returns a vector of all the words. The istringstream code I have below works fine except for when there is punctuation so naturally my fix is to want to "sanitize" the line before I run it through the while loop.

I would appreciate some help in using the regex library in c++ for this. My initial solution was to us substr() and go to town but that seems complicated as I'll have to iterate and test each character to see what it is then perform some operations.

vector<string> lineParser(Line * ln)
{
    vector<string> result;
    string word;
    string line = ln->getLine();
    istringstream iss(line);
    while(iss)
    {
        iss >> word;
        result.push_back(word);
    }
    return result;
}
7
  • You should probably specify which regex library for c++ you want to use. There is no regex in STL - are you using this: boost.org/doc/libs/1_46_1/libs/regex/doc/html/index.html?
    – Jamie Wong
    Apr 4, 2011 at 14:35
  • I am using #include <regex> in Visual Studio 2010. I didn't install anything special and thought it was part of the STL. If that's not the case then I don't know.
    – Pete
    Apr 4, 2011 at 14:39
  • 1
    <regex> is new in C++11. C++11 was just voted out a couple weeks ago, and is just shy of being official. VC++2010 supports this new C++11 feature. Apr 4, 2011 at 14:44
  • "I'll have to iterate and test each character to see what it is then perform some operations.", and the regex code won't?
    – genpfault
    Apr 4, 2011 at 14:50
  • 1
    s/[^\d\s]// is a lot shorter than the C++ equivalent.
    – tstenner
    Apr 4, 2011 at 14:52

4 Answers 4

7

Don't need to use regular expressions just for punctuation:

// Replace all punctuation with space character.
std::replace_if(line.begin(), line.end(),
                std::ptr_fun<int, int>(&std::ispunct),
                ' '
               );

Or if you want everything but letters and numbers turned into space:

std::replace_if(line.begin(), line.end(),
                std::not1(std::ptr_fun<int,int>(&std::isalphanum)),
                ' '
               );

While we are here:
Your while loop is broken and will push the last value into the vector twice.

It should be:

while(iss)
{
    iss >> word;
    if (iss)                    // If the read of a word failed. Then iss state is bad.
    {    result.push_back(word);// Only push_back() if the state is not bad.
    }
}

Or the more common version:

while(iss >> word) // Loop is only entered if the read of the word worked.
{
    result.push_back(word);
}

Or you can use the stl:

std::copy(std::istream_iterator<std::string>(iss),
          std::istream_iterator<std::string>(),
          std::back_inserter(result)
         );
2
  • btw, ispunct method is about 10x faster than boost::regex, on my machine Aug 21, 2015 at 16:49
  • That is because ispuct is simply an array lookup. Aug 21, 2015 at 17:08
2

[^A-Za-z\s] should do what you need if your replace the matching characters by nothing. It should remove all characters that are not letters and spaces. Or [^A-Za-z0-9\s] if you want to keep numbers too.

You can use online tools like this one : http://gskinner.com/RegExr/ to test out your patterns (Replace tab). Indeed some modifications can be required based on the regex lib you are using.

2
  • I ended up using a variant ([^A-Za-z0-9\\s\\']) of this regular expression that also includes apostrophes for words like "I've" and "didn't". The link was also very helpful.
    – Pete
    Apr 12, 2011 at 5:20
  • with [^A-Za-z\s], compilers warns me unknown escape sequence: '\s' [enabled by default]. Escaping s twice solved the problem. Aug 21, 2015 at 15:43
1

I'm not positive, but I think this is what you're looking for:

#include<iostream>
#include<regex>
#include<vector>

int
main()
{
    std::string line("some words: with some punctuation.");
    std::regex words("[\\w]+");
    std::sregex_token_iterator i(line.begin(), line.end(), words);
    std::vector<std::string> list(i, std::sregex_token_iterator());
    for (auto j = list.begin(), e = list.end(); j != e; ++j)
        std::cout << *j << '\n';
}

some
words
with
some
punctuation
0

The simplest solution is probably to create a filtering streambuf to convert all non alphanumeric characters to space, then to read using std::copy:

class StripPunct : public std::streambuf
{
    std::streambuf* mySource;
    char            myBuffer;

protected:
    virtual int underflow()
    {
        int result = mySource->sbumpc();
        if ( result != EOF ) {
            if ( !::isalnum( result ) )
                result = ' ';
            myBuffer = result;
            setg( &myBuffer, &myBuffer, &myBuffer + 1 );
        }
        return result;
    }

public:
    explicit StripPunct( std::streambuf* source )
        : mySource( source )
    {
    }
};

std::vector<std::string>
LineParser( std::istream& source )
{
    StripPunct               sb( source.rdbuf() );
    std::istream             src( &sb );
    return std::vector<std::string>(
        (std::istream_iterator<std::string>( src )),
        (std::istream_iterator<std::string>()) );
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.