5

I'm making a crawler. User can specify regular expression string to download data.

When user input form is:

http://xxx/abc[x-z]/image(9|10|11).png

I want to download these.

http://xxx/abcx/image9.png
http://xxx/abcy/image9.png
http://xxx/abcz/image9.png
http://xxx/abcx/image10.png
http://xxx/abcy/image10.png
http://xxx/abcz/image10.png
http://xxx/abcx/image11.png
http://xxx/abcy/image11.png
http://xxx/abcz/image11.png

Can I create the following list from the above regular expression string? Or, can I use each string in for-in block?

11
  • Add more details, why you want these and what you tried so far..It would help us to find better and quick solution
    – Learner
    Nov 20, 2015 at 13:18
  • you could use loop here Nov 20, 2015 at 13:18
  • 1
    Your regular expression is invalid, since [9-11] is not a valid character class. We would need more context on what you are actually trying to achieve to be able to help you. Nov 20, 2015 at 13:20
  • Thank you for a lot of advice. I added details. And I try to use product(). Nov 20, 2015 at 13:39
  • @SvenMarnach Thank you, I check regular expressions rule :) Nov 20, 2015 at 13:42

3 Answers 3

10

If you are wanting to take a user's given regex as an input and generate a list of strings you can use the library sre_yield:

However, be very aware that trying to parse every possible string of a regex can get out of hand very quickly. You'll need to be sure that your users are aware of the implications that wildcard characters and open ended or repeating groups can have on the number of possible matching strings.

As an example, your regex string: http://xxx/abc[x-z]/image(9|10|11).png does not escape the ., which is a wildcard for any character, so it will generate a lot of unexpected strings. Instead we'll need to escape it as seen in the example below:

>>> import sre_yield

>>> links = []

>>> for each in sre_yield.AllStrings(r'http://xxx/abc[x-z]/image(9|10|11)\.png'):
        links.append(each)

Or more simply links = list(sre_yield.AllStrings(r'http://xxx/abc[x-z]/image(9|10|11)\.png'))

The result is:

>>> links

['http://xxx/abcx/image9.png', 'http://xxx/abcy/image9.png', 
'http://xxx/abcz/image9.png', 'http://xxx/abcx/image10.png', 
'http://xxx/abcy/image10.png', 'http://xxx/abcz/image10.png', 
'http://xxx/abcx/image11.png', 'http://xxx/abcy/image11.png', 
'http://xxx/abcz/image11.png']
1
  • 2
    Wow, I tried sre_yield. That's perfect. I was able to achieve my purpose. Thank you for your kindness, and I'll take care to .. Nov 21, 2015 at 5:51
1

You can use product() from the itertools builtin:

from itertools import product

for x, y in product(['x', 'y', 'z'], range(9, 12)):
    print 'http://xxx/abc{}/image{}'.format(x, y)

To build your list you can use a comprehension:

links = ['http://xxx/abc{}/image{}'.format(x, y) for x, y in product(['x', 'y', 'z'], range(9, 12))]
9
  • You could, but (a) this doesn't answer the question as written, since it's not creating the list of strings from a regular expression and (b) a nested loop would be much more straight-forward here in my opinion. Nov 20, 2015 at 13:21
  • I believe it is useful anyway, because (a) the op does not have a regular expression anyway, from which I guess he is building his own thing, this answer providing a major building block and (b) the cross-product tool is much more versatile, expressive and extensible than nested loops.
    – spectras
    Nov 20, 2015 at 13:41
  • @spectras: I disagree about (b). itertools.product() is heavily overused, and in almost all cases it reduces readability, since it conceals the link between the individual loop variables and the iterables they iterate over and unnecessarily increases verbosity. Nov 20, 2015 at 13:46
  • (Cannot edit my previous comment, but I obviously meant cartesian product, not cross product). I disagree though. If you are used to dealing with functional programming, product is much more natural than loops. What matters is the output, not the intermediate loop variables. To me, product(a, b, c) is much more readable than a triple loop with intermediate variables and 2 additional levels of nesting. But I guess it's a matter of style.
    – spectras
    Nov 20, 2015 at 13:50
  • @spectras So you are basically saying all nested loops should be written using itertools.product(). I like Haskell, I am used to functional programming, but the context here is Python, so there is no way around the loop variables. Giving a different out-of-context example doesn't really help the argument that itertools.product() should be used here either. Nov 20, 2015 at 13:57
0

Simple try may be-alternative to the previous answers

lst = ['http://xxx/abc%s/image%s.png'%(x,y) for x, y in [(j,i) for i in (9,10,11) for j in ('x', 'y', 'z')]]

Omitted range and format function for quicker performance.


Analysis- I compared my way and the way posted by Jkdc

I ran both way 100000 times but mean shows that itertools approach is faster in terms of execution time-

from itertools import product
import time
from matplotlib import pyplot as plt
import numpy as np

prodct = []
native = []

def test():
    start = time.clock()
    lst = ['http://xxx/abc{}/image{}'.format(x, y) for x, y in product(('x', 'y', 'z'), range(9, 11))]
    end = time.clock()

    print '{0:.50f}'.format(end-start)
    prodct.append('{0:.50f}'.format(end-start))

    start1 = time.clock()
    lst = ['http://xxx/abc%s/image%s'%(x,y) for x, y in [(j,i) for i in (9,10,11) for j in ('x', 'y', 'z')]]
    end1 = time.clock()

    print '{0:.50f}'.format(end1-start1)
    native.append('{0:.50f}'.format(end1-start1))


for i in range(1,100000):
    test()


y = np.dot(np.array(native).astype(np.float),100000)
x= np.dot(np.array(prodct).astype(np.float),100000)

print np.mean(y)
print np.mean(x)

graph and getting result for native(no module) and itertools-product as below

for native 2.1831179834 for itertools-product 1.60410432562

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.