358

Here's the simplest way to explain this. Here's what I'm using:

re.split('\W', 'foo/bar spam\neggs')
>>> ['foo', 'bar', 'spam', 'eggs']

Here's what I want:

someMethod('\W', 'foo/bar spam\neggs')
>>> ['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs']

The reason is that I want to split a string into tokens, manipulate it, then put it back together again.

3

21 Answers 21

465

The docs of re.split mention:

Split string by the occurrences of pattern. If capturing parentheses are used in pattern, then the text of all groups in the pattern are also returned as part of the resulting list.

So you just need to wrap your separator with a capturing group:

>>> re.split('(\W)', 'foo/bar spam\neggs')
['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs']
11
  • 34
    That's cool. I didn't know re.split did that with capture groups. Jan 25, 2010 at 23:48
  • 28
    @Laurence: Well, it's documented: docs.python.org/library/re.html#re.split: "Split string by the occurrences of pattern. If capturing parentheses are used in pattern, then the text of all groups in the pattern are also returned as part of the resulting list." Jan 25, 2010 at 23:54
  • 53
    It's seriously underdocumented. I've been using Python for 14 years and only just found this out.
    – smci
    Jun 19, 2013 at 16:33
  • 34
    Is there an option so that the output of the group match is attached to whatever is on the left (or analogously right) of the split? For example, can this be easily modified so the output is ['foo', '/bar', ' spam', '\neggs']?
    – ely
    Feb 9, 2015 at 2:24
  • 4
    @Mr.F You might be able to do something with re.sub. I wanted to split on a ending percent so I just subbed in a double character and then split, hacky but worked for my case: re.split('% ', re.sub('% ', '%% ', '5.000% Additional Whatnot')) --> ['5.000%', 'Additional Whatnot'] Oct 11, 2015 at 22:41
54

If you are splitting on newline, use splitlines(True).

>>> 'line 1\nline 2\nline without newline'.splitlines(True)
['line 1\n', 'line 2\n', 'line without newline']

(Not a general solution, but adding this here in case someone comes here not realizing this method existed.)

0
38

Another example: split on non alpha-numeric and keep the separators

import re
a = "foo,bar@candy*ice%cream"
re.split('([^a-zA-Z0-9])', a)

Output:

['foo', ',', 'bar', '@', 'candy', '*', 'ice', '%', 'cream']

Explanation re.split('([^a-zA-Z0-9])', a):

() <- keep the separators
[] <- match everything in between
^a-zA-Z0-9 <- except alphabets, upper/lower and numbers.
3
  • 1
    Even though, as the docs say, this is equivalent to the accepted answer, I like this version's readability--even though \W is a more compact way to express it.
    – ephsmith
    Oct 17, 2018 at 1:03
  • I like the readability of this as well, plus you can customize it if you want to include/exclude some chars!
    – tikka
    Dec 11, 2020 at 15:04
  • 1
    it's also better for "crazy" langauges which use punctuations as part of a word. Some Hebrew words has ' or " built in (כפר אז"ר, ג'ירף), which requires special treatment. May 6, 2021 at 15:57
19

If you have only 1 separator, you can employ list comprehensions:

text = 'foo,bar,baz,qux'  
sep = ','

Appending/prepending separator:

result = [x+sep for x in text.split(sep)]
#['foo,', 'bar,', 'baz,', 'qux,']
# to get rid of trailing
result[-1] = result[-1].strip(sep)
#['foo,', 'bar,', 'baz,', 'qux']

result = [sep+x for x in text.split(sep)]
#[',foo', ',bar', ',baz', ',qux']
# to get rid of trailing
result[0] = result[0].strip(sep)
#['foo', ',bar', ',baz', ',qux']

Separator as it's own element:

result = [u for x in text.split(sep) for u in (x, sep)]
#['foo', ',', 'bar', ',', 'baz', ',', 'qux', ',']
results = result[:-1]   # to get rid of trailing
2
  • 2
    you can also add in if x to ensure that the chunk produced by split has some content, i.e. result = [x + sep for x in text.split(sep) if x] May 8, 2020 at 15:12
  • For me, strip removed too much and I had to use this: result = [sep+x for x in data.split(sep)] result[0] = result[0][len(sep):] Jun 10, 2020 at 0:49
12

Another no-regex solution that works well on Python 3

# Split strings and keep separator
test_strings = ['<Hello>', 'Hi', '<Hi> <Planet>', '<', '']

def split_and_keep(s, sep):
   if not s: return [''] # consistent with string.split()

   # Find replacement character that is not used in string
   # i.e. just use the highest available character plus one
   # Note: This fails if ord(max(s)) = 0x10FFFF (ValueError)
   p=chr(ord(max(s))+1) 

   return s.replace(sep, sep+p).split(p)

for s in test_strings:
   print(split_and_keep(s, '<'))


# If the unicode limit is reached it will fail explicitly
unicode_max_char = chr(1114111)
ridiculous_string = '<Hello>'+unicode_max_char+'<World>'
print(split_and_keep(ridiculous_string, '<'))
0
8

One lazy and simple solution

Assume your regex pattern is split_pattern = r'(!|\?)'

First, you add some same character as the new separator, like '[cut]'

new_string = re.sub(split_pattern, '\\1[cut]', your_string)

Then you split the new separator, new_string.split('[cut]').

2
  • This approach is clever, but will fail when the original string already contains [cut] somewhere. Sep 19, 2019 at 9:27
  • It could be faster on large scale problems as it finally uses string.split(), in case that re.split() costs more than re.sub() with string.split() (which I do not know). Jun 26, 2020 at 12:43
7

Here is a simple .split solution that works without regex.

This is an answer for Python split() without removing the delimiter, so not exactly what the original post asks but the other question was closed as a duplicate for this one.

def splitkeep(s, delimiter):
    split = s.split(delimiter)
    return [substr + delimiter for substr in split[:-1]] + [split[-1]]

Random tests:

import random

CHARS = [".", "a", "b", "c"]
assert splitkeep("", "X") == [""]  # 0 length test
for delimiter in ('.', '..'):
    for _ in range(100000):
        length = random.randint(1, 50)
        s = "".join(random.choice(CHARS) for _ in range(length))
        assert "".join(splitkeep(s, delimiter)) == s
1
  • regex should be avoided on large scale problems for speed reasons, that is why this is a good hint. Jun 26, 2020 at 12:40
5

You can also split a string with an array of strings instead of a regular expression, like this:

def tokenizeString(aString, separators):
    #separators is an array of strings that are being used to split the string.
    #sort separators in order of descending length
    separators.sort(key=len)
    listToReturn = []
    i = 0
    while i < len(aString):
        theSeparator = ""
        for current in separators:
            if current == aString[i:i+len(current)]:
                theSeparator = current
        if theSeparator != "":
            listToReturn += [theSeparator]
            i = i + len(theSeparator)
        else:
            if listToReturn == []:
                listToReturn = [""]
            if(listToReturn[-1] in separators):
                listToReturn += [""]
            listToReturn[-1] += aString[i]
            i += 1
    return listToReturn
    

print(tokenizeString(aString = "\"\"\"hi\"\"\" hello + world += (1*2+3/5) '''hi'''", separators = ["'''", '+=', '+', "/", "*", "\\'", '\\"', "-=", "-", " ", '"""', "(", ")"]))
4
  1. replace all seperator: (\W) with seperator + new_seperator: (\W;)

  2. split by the new_seperator: (;)

def split_and_keep(seperator, s):
  return re.split(';', re.sub(seperator, lambda match: match.group() + ';', s))

print('\W', 'foo/bar spam\neggs')
1
  • Yeah this is better, although switching the order of when ';' is appended worked. Mar 14, 2021 at 17:21
3
# This keeps all separators  in result 
##########################################################################
import re
st="%%(c+dd+e+f-1523)%%7"
sh=re.compile('[\+\-//\*\<\>\%\(\)]')

def splitStringFull(sh, st):
   ls=sh.split(st)
   lo=[]
   start=0
   for l in ls:
     if not l : continue
     k=st.find(l)
     llen=len(l)
     if k> start:
       tmp= st[start:k]
       lo.append(tmp)
       lo.append(l)
       start = k + llen
     else:
       lo.append(l)
       start =llen
   return lo
  #############################

li= splitStringFull(sh , st)
['%%(', 'c', '+', 'dd', '+', 'e', '+', 'f', '-', '1523', ')%%', '7']
2

If one wants to split string while keeping separators by regex without capturing group:

def finditer_with_separators(regex, s):
    matches = []
    prev_end = 0
    for match in regex.finditer(s):
        match_start = match.start()
        if (prev_end != 0 or match_start > 0) and match_start != prev_end:
            matches.append(s[prev_end:match.start()])
        matches.append(match.group())
        prev_end = match.end()
    if prev_end < len(s):
        matches.append(s[prev_end:])
    return matches

regex = re.compile(r"[\(\)]")
matches = finditer_with_separators(regex, s)

If one assumes that regex is wrapped up into capturing group:

def split_with_separators(regex, s):
    matches = list(filter(None, regex.split(s)))
    return matches

regex = re.compile(r"([\(\)])")
matches = split_with_separators(regex, s)

Both ways also will remove empty groups which are useless and annoying in most of the cases.

1
  • 1
    This ended up working perfectly for me. Thanks for your contribution! Jan 7, 2022 at 15:40
2

install wrs "WITHOUT REMOVING SPLITOR" BY DOING

pip install wrs

(developed by Rao Hamza)

import wrs
text  = "Now inbox “how to make spam ad” Invest in hard email marketing."
splitor = 'email | spam | inbox'
list = wrs.wr_split(splitor, text)
print(list)

result: ['now ', 'inbox “how to make ', 'spam ad” invest in hard ', 'email marketing.']

1
1

I had a similar issue trying to split a file path and struggled to find a simple answer. This worked for me and didn't involve having to substitute delimiters back into the split text:

my_path = 'folder1/folder2/folder3/file1'

import re

re.findall('[^/]+/|[^/]+', my_path)

returns:

['folder1/', 'folder2/', 'folder3/', 'file1']

2
  • This can be slightly simplified by using: re.findall('[^/]+/?', my_path) (e.g. making the trailing slash optional using a ? rather than providing two alternatives with |. Sep 19, 2019 at 9:30
  • for paths, you're far better off using the stdlib os.path functions
    – anon01
    Aug 21, 2020 at 6:31
1

May I just leave it here

s = 'foo/bar spam\neggs'
print(s.replace('/', '+++/+++').replace(' ', '+++ +++').replace('\n', '+++\n+++').split('+++'))

['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs']
1

How can we split a string in Python, including whitespace or condinuoes whitespace?

def splitWithSpace(string):
    list_strings = list(string)
    split_list = []
    new_word = ""
    for characters in list_strings:
        if character == " ":
            split_list.extend([new_word, " "]) if new_word  else split_list.append(" ")
            new_word = ""
        else:
            new_word += character
    split_list.append(new_word)
    print(split_list)

Single Space:

splitWithSpace("this is a simple text")

Answer: ['this', ' ', 'is', ' ', 'a', ' ', 'simple', ' ', 'text']

More Space:

splitWithSpace("this is  a  simple text")

Answer: ['this', ' ', 'is', ' ', ' ', 'a', ' ', ' ', 'simple', ' ', 'text']

1
0

I found this generator based approach more satisfying:

def split_keep(string, sep):
    """Usage:
    >>> list(split_keep("a.b.c.d", "."))
    ['a.', 'b.', 'c.', 'd']
    """
    start = 0
    while True:
        end = string.find(sep, start) + 1
        if end == 0:
            break
        yield string[start:end]
        start = end
    yield string[start:]

It avoids the need to figure out the correct regex, while in theory should be fairly cheap. It doesn't create new string objects and, delegates most of the iteration work to the efficient find method.

... and in Python 3.8 it can be as short as:

def split_keep(string, sep):
    start = 0
    while (end := string.find(sep, start) + 1) > 0:
        yield string[start:end]
        start = end
    yield string[start:]
0

Some of those answers posted before, will repeat delimiter, or have some other bugs which I faced in my case. You can use this function, instead:

def split_and_keep_delimiter(input, delimiter):
    result      = list()
    idx         = 0
    while delimiter in input:
        idx     = input.index(delimiter);
        result.append(input[0:idx+len(delimiter)])
        input = input[idx+len(delimiter):]
    result.append(input)
    return result
1
  • 1
    This function is incorrect - it sometimes returns an empty string at the end. Try test_splitter('Hello World ! ',' ') and you get ['Hello ', 'World ', ' ', '! ', ' ', ' ', ''] Aug 31, 2022 at 20:43
0

In the below code, there is a simple, very efficient and well tested answer to this question. The code has comments explaining everything in it.

I promise it's not as scary as it looks - it's actually only 13 lines of code! The rest are all comments, docs and assertions

def split_including_delimiters(input: str, delimiter: str):
    """
    Splits an input string, while including the delimiters in the output
    
    Unlike str.split, we can use an empty string as a delimiter
    Unlike str.split, the output will not have any extra empty strings
    Conequently, len(''.split(delimiter))== 0 for all delimiters,
       whereas len(input.split(delimiter))>0 for all inputs and delimiters
    
    INPUTS:
        input: Can be any string
        delimiter: Can be any string

    EXAMPLES:
         >>> split_and_keep_delimiter('Hello World  ! ',' ')
        ans = ['Hello ', 'World ', ' ', '! ', ' ']
         >>> split_and_keep_delimiter("Hello**World**!***", "**")
        ans = ['Hello', '**', 'World', '**', '!', '**', '*']
    EXAMPLES:
        assert split_and_keep_delimiter('-xx-xx-','xx') == ['-', 'xx', '-', 'xx', '-'] # length 5
        assert split_and_keep_delimiter('xx-xx-' ,'xx') == ['xx', '-', 'xx', '-']      # length 4
        assert split_and_keep_delimiter('-xx-xx' ,'xx') == ['-', 'xx', '-', 'xx']      # length 4
        assert split_and_keep_delimiter('xx-xx'  ,'xx') == ['xx', '-', 'xx']           # length 3
        assert split_and_keep_delimiter('xxxx'   ,'xx') == ['xx', 'xx']                # length 2
        assert split_and_keep_delimiter('xxx'    ,'xx') == ['xx', 'x']                 # length 2
        assert split_and_keep_delimiter('x'      ,'xx') == ['x']                       # length 1
        assert split_and_keep_delimiter(''       ,'xx') == []                          # length 0
        assert split_and_keep_delimiter('aaa'    ,'xx') == ['aaa']                     # length 1
        assert split_and_keep_delimiter('aa'     ,'xx') == ['aa']                      # length 1
        assert split_and_keep_delimiter('a'      ,'xx') == ['a']                       # length 1
        assert split_and_keep_delimiter(''       ,''  ) == []                          # length 0
        assert split_and_keep_delimiter('a'      ,''  ) == ['a']                       # length 1
        assert split_and_keep_delimiter('aa'     ,''  ) == ['a', '', 'a']              # length 3
        assert split_and_keep_delimiter('aaa'    ,''  ) == ['a', '', 'a', '', 'a']     # length 5
    """

    # Input assertions
    assert isinstance(input,str), "input must be a string"
    assert isinstance(delimiter,str), "delimiter must be a string"

    if delimiter:
        # These tokens do not include the delimiter, but are computed quickly
        tokens = input.split(delimiter)
    else:
        # Edge case: if the delimiter is the empty string, split between the characters
        tokens = list(input)
        
    # The following assertions are always true for any string input and delimiter
    # For speed's sake, we disable this assertion
    # assert delimiter.join(tokens) == input

    output = tokens[:1]

    for token in tokens[1:]:
        output.append(delimiter)
        if token:
            output.append(token)
    
    # Don't let the first element be an empty string
    if output[:1]==['']:
        del output[0]
        
    # The only case where we should have an empty string in the output is if it is our delimiter
    # For speed's sake, we disable this assertion
    # assert delimiter=='' or '' not in output
        
    # The resulting strings should be combinable back into the original string
    # For speed's sake, we disable this assertion
    # assert ''.join(output) == input

    return output
0
>>> line = 'hello_toto_is_there'
>>> sep = '_'
>>> [sep + x[1] if x[0] != 0 else x[1] for x in enumerate(line.split(sep))]
['hello', '_toto', '_is', '_there']
0

An implementation that uses only list (with help of str.partition()):

import typing as t


def partition(s: str, seps: t.Iterable[str]):
    if not s or not seps:
        return [s]
    st1, st2 = [s], []
    for sep in set(seps):
        if st1:
            while st1:
                st2.append(st1.pop())
                while True:
                    x1, x2, x3 = st2.pop().rpartition(sep)
                    if not x2:  # `sep` not found
                        st2.append(x3)
                        break
                    if not x1:
                        st2.extend([x3, x2] if x3 else [x2])
                        break
                    st2.extend([x3, x2, x1] if x3 else [x2, x1])
        else:
            while st2:
                st1.append(st2.pop())
                while True:
                    x1, x2, x3 = st1.pop().partition(sep)
                    if not x2:  # `sep` not found
                        st1.append(x1)
                        break
                    if not x3:
                        st1.extend([x1, x2] if x1 else [x2])
                        break
                    st1.extend([x1, x2, x3] if x1 else [x2, x3])
    return st1 or list(reversed(st2))
assert partition('abcdbcd', ['a']) == ['a', 'bcdbcd']
assert partition('abcdbcd', ['b']) == ['a', 'b', 'cd', 'b', 'cd']
assert partition('abcdbcd', ['d']) == ['abc', 'd', 'bc', 'd']
assert partition('abcdbcd', ['e']) == ['abcdbcd']
assert partition('abcdbcd', ['b', 'd']) == ['a', 'b', 'c', 'd', 'b', 'c', 'd']
assert partition('abcdbcd', ['db']) == ['abc', 'db', 'cd']
0

Use re.split. Also your regular expression comes from a variable and also you have a multi separator. You can use as the following:

# BashSpecialParamList is the special parameter in Bash,
# such as your separator is the Bash special parameter
BashSpecialParamList = ["$*", "$@", "$#", "$?", "$-", "$$", "$!", "$0"]
# aStr is the the string to be split
aStr = "$a Klkjfd$0 $? $#%$*Sdfdf"

reStr = "|".join([re.escape(sepStr) for sepStr in BashSpecialParamList])

re.split(f'({reStr})', aStr)

# Then you can get the result:
# ['$a Klkjfd', '$0', ' ', '$?', ' ', '$#', '%', '$*', 'Sdfdf']

Reference: GNU Bash Special Parameters

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.