2

I am generating all combinations from a set of numbers, and then want to generate combinations of those combinations. Because of the vast number of possible combinations I keep getting a memory error. I have looked at the following questions but none have really solved my problem:
Creating all combinations of a set and running out of memory
Python itertools.combinations() memory problems
Python list memory error

I am generating my list using the following method:

#generate all combinations of 1 and 0 of size 30
set_1 = itertools.product([0,1], repeat = 30)
#generate all combinations of set 1, of size 5
set_2 = [tuple(c) for c in pulp.allcombinations(set_1, 5)]
for sets in set_2:
    print(sets)

Them memory error occurs while it is generating set_2. I would like to still be able to iterate over set_2 as I will need to access the sets later on. I have considered writing the sets to a txt file, but I would like to save that as a last resort.

1 Answer 1

1

Instead of using a list comprehension which creates a list in your memory you can use a generator expression to store set2 and save your memory:

set_2 = (tuple(c) for c in pulp.allcombinations(set_1, 5))

Generators are like list comprehension except that they doesn't store the values in memory and just produce the values on demand.But they are one shot iterators and you can not iterate over them again like the result of a list comprehension.

3
  • Thankks, this fixed the set generation problem. Unfortunatly there is another part in my code where I create an LpVariable like this: ` x = pulp.LpVariable.dicts("sets", set_2, lowBound = 0, upBound = 1, cat = pulp.LpInteger) ` which now throws a memory error
    – user35510
    Jul 15, 2015 at 14:19
  • Can you make pulp.LpVariable.dicts return a generator?
    – Mazdak
    Jul 15, 2015 at 14:22
  • I don't think so, I need to create a dictionary to tell me which sets are being used in the overall solution to the LP problem.
    – user35510
    Jul 15, 2015 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.