How would one create an iterative function (or iterator object) in python?
10 Answers
Iterator objects in python conform to the iterator protocol, which basically means they provide two methods: __iter__() and __next__().
The
__iter__returns the iterator object and is implicitly called at the start of loops.The
__next__()method returns the next value and is implicitly called at each loop increment. This method raises a StopIteration exception when there are no more value to return, which is implicitly captured by looping constructs to stop iterating.
Here's a simple example of a counter:
class Counter:
def __init__(self, low, high):
self.current = low - 1
self.high = high
def __iter__(self):
return self
def __next__(self): # Python 2: def next(self)
self.current += 1
if self.current < self.high:
return self.current
raise StopIteration
for c in Counter(3, 9):
print(c)
This will print:
3
4
5
6
7
8
This is easier to write using a generator, as covered in a previous answer:
def counter(low, high):
current = low
while current < high:
yield current
current += 1
for c in counter(3, 9):
print(c)
The printed output will be the same. Under the hood, the generator object supports the iterator protocol and does something roughly similar to the class Counter.
David Mertz's article, Iterators and Simple Generators, is a pretty good introduction.
-
8This is mostly a good answer, but the fact that it returns self is a little sub-optimal. For example, if you used the same counter object in a doubly nested for loop you would probably not get the behavior that you meant. Feb 6, 2014 at 23:33
-
37No, iterators SHOULD return themselves. Iterables return iterators, but iterables shouldn't implement
__next__.counteris an iterator, but it is not a sequence. It doesn't store its values. You shouldn't be using the counter in a doubly-nested for-loop, for example.– leewzFeb 21, 2014 at 8:42 -
5In the Counter example, self.current should be assigned in
__iter__(in addition to in__init__). Otherwise, the object can be iterated only once. E.g., if you sayctr = Counters(3, 8), then you cannot usefor c in ctrmore than once.– CurtApr 5, 2016 at 23:00 -
10@Curt: Absolutely not.
Counteris an iterator, and iterators are only supposed to be iterated once. If you resetself.currentin__iter__, then a nested loop over theCounterwould be completely broken, and all sorts of assumed behaviors of iterators (that callingiteron them is idempotent) are violated. If you want to be able to iteratectrmore than once, it needs to be a non-iterator iterable, where it returns a brand new iterator each time__iter__is invoked. Trying to mix and match (an iterator that is implicitly reset when__iter__is invoked) violates the protocols. Feb 24, 2018 at 1:16 -
3For example, if
Counterwas to be a non-iterator iterable, you'd remove the definition of__next__/nextentirely, and probably redefine__iter__as a generator function of the same form as the generator described at the end of this answer (except instead of the bounds coming from arguments to__iter__, they'd be arguments to__init__saved onselfand accessed fromselfin__iter__). Feb 24, 2018 at 1:19
There are four ways to build an iterative function:
- create a generator (uses the yield keyword)
- use a generator expression (genexp)
- create an iterator (defines
__iter__and__next__(ornextin Python 2.x)) - create a class that Python can iterate over on its own (defines
__getitem__)
Examples:
# generator
def uc_gen(text):
for char in text.upper():
yield char
# generator expression
def uc_genexp(text):
return (char for char in text.upper())
# iterator protocol
class uc_iter():
def __init__(self, text):
self.text = text.upper()
self.index = 0
def __iter__(self):
return self
def __next__(self):
try:
result = self.text[self.index]
except IndexError:
raise StopIteration
self.index += 1
return result
# getitem method
class uc_getitem():
def __init__(self, text):
self.text = text.upper()
def __getitem__(self, index):
return self.text[index]
To see all four methods in action:
for iterator in uc_gen, uc_genexp, uc_iter, uc_getitem:
for ch in iterator('abcde'):
print(ch, end=' ')
print()
Which results in:
A B C D E
A B C D E
A B C D E
A B C D E
Note:
The two generator types (uc_gen and uc_genexp) cannot be reversed(); the plain iterator (uc_iter) would need the __reversed__ magic method (which, according to the docs, must return a new iterator, but returning self works (at least in CPython)); and the getitem iteratable (uc_getitem) must have the __len__ magic method:
# for uc_iter we add __reversed__ and update __next__
def __reversed__(self):
self.index = -1
return self
def __next__(self):
try:
result = self.text[self.index]
except IndexError:
raise StopIteration
self.index += -1 if self.index < 0 else +1
return result
# for uc_getitem
def __len__(self)
return len(self.text)
To answer Colonel Panic's secondary question about an infinite lazily evaluated iterator, here are those examples, using each of the four methods above:
# generator
def even_gen():
result = 0
while True:
yield result
result += 2
# generator expression
def even_genexp():
return (num for num in even_gen()) # or even_iter or even_getitem
# not much value under these circumstances
# iterator protocol
class even_iter():
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
next_value = self.value
self.value += 2
return next_value
# getitem method
class even_getitem():
def __getitem__(self, index):
return index * 2
import random
for iterator in even_gen, even_genexp, even_iter, even_getitem:
limit = random.randint(15, 30)
count = 0
for even in iterator():
print even,
count += 1
if count >= limit:
break
print
Which results in (at least for my sample run):
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
How to choose which one to use? This is mostly a matter of taste. The two methods I see most often are generators and the iterator protocol, as well as a hybrid (__iter__ returning a generator).
Generator expressions are useful for replacing list comprehensions (they are lazy and so can save on resources).
If one needs compatibility with earlier Python 2.x versions use __getitem__.
-
4I like this summary because it is complete. Those three ways (yield, generator expression and iterator) are essentially the same, although some are more convenient than others. The yield operator captures the "continuation" which contains the state (for example the index that we are up to). The information is saved in the "closure" of the continuation. The iterator way saves the same information inside the fields of the iterator, which is essentially the same thing as a closure. The getitem method is a little different because it indexes into the contents and is not iterative in nature.– IanJul 5, 2013 at 1:04
-
2@metaperl: Actually, it is. In all four of the above cases you can use the same code to iterate. Nov 5, 2013 at 16:37
-
1@Asterisk: No, an instance of
uc_itershould expire when it's done (otherwise it would by infinite); if you want to do it again you have to get a new iterator by callinguc_iter()again. Apr 19, 2018 at 16:13 -
3You can set
self.index = 0in__iter__so that you can iterate many times over. Otherwise you can't. Aug 14, 2018 at 8:26 -
1If you could spare the time I would appreciate an explanation for why you would choose any of the methods over the others.– aaaaaaJan 21, 2019 at 4:15
I see some of you doing return self in __iter__. I just wanted to note that __iter__ itself can be a generator (thus removing the need for __next__ and raising StopIteration exceptions)
class range:
def __init__(self,a,b):
self.a = a
self.b = b
def __iter__(self):
i = self.a
while i < self.b:
yield i
i+=1
Of course here one might as well directly make a generator, but for more complex classes it can be useful.
-
5Great! It so boring writing just
return selfin__iter__. When I was going to try usingyieldin it I found your code doing exactly what I want to try.– RayFeb 5, 2013 at 19:32 -
3But in this case, how would one implement
next()?return iter(self).next()?– LennaApr 5, 2013 at 19:52 -
4@Lenna, it is already "implemented" because iter(self) returns an iterator, not a range instance.– ManuxApr 7, 2013 at 17:31
-
3This the easiest way of doing it, and doesn't involve having to keep track of e.g.
self.currentor any other counter. This should be the top-voted answer! Mar 31, 2014 at 13:35 -
14To be clear, this approach makes your class iterable, but not an iterator. You get fresh iterators every time you call
iteron instances of the class, but they're not themselves instances of the class. Feb 24, 2018 at 1:25
First of all the itertools module is incredibly useful for all sorts of cases in which an iterator would be useful, but here is all you need to create an iterator in python:
yield
Isn't that cool? Yield can be used to replace a normal return in a function. It returns the object just the same, but instead of destroying state and exiting, it saves state for when you want to execute the next iteration. Here is an example of it in action pulled directly from the itertools function list:
def count(n=0):
while True:
yield n
n += 1
As stated in the functions description (it's the count() function from the itertools module...) , it produces an iterator that returns consecutive integers starting with n.
Generator expressions are a whole other can of worms (awesome worms!). They may be used in place of a List Comprehension to save memory (list comprehensions create a list in memory that is destroyed after use if not assigned to a variable, but generator expressions can create a Generator Object... which is a fancy way of saying Iterator). Here is an example of a generator expression definition:
gen = (n for n in xrange(0,11))
This is very similar to our iterator definition above except the full range is predetermined to be between 0 and 10.
I just found xrange() (suprised I hadn't seen it before...) and added it to the above example. xrange() is an iterable version of range() which has the advantage of not prebuilding the list. It would be very useful if you had a giant corpus of data to iterate over and only had so much memory to do it in.
-
21as of python 3.0 there is no longer an xrange() and the new range() behaves like the old xrange()– user3850Dec 18, 2008 at 17:30
-
6You should still use xrange in 2._, because 2to3 translates it automatically.– PhobJul 22, 2011 at 18:03
This question is about iterable objects, not about iterators. In Python, sequences are iterable too so one way to make an iterable class is to make it behave like a sequence, i.e. give it __getitem__ and __len__ methods. I have tested this on Python 2 and 3.
class CustomRange:
def __init__(self, low, high):
self.low = low
self.high = high
def __getitem__(self, item):
if item >= len(self):
raise IndexError("CustomRange index out of range")
return self.low + item
def __len__(self):
return self.high - self.low
cr = CustomRange(0, 10)
for i in cr:
print(i)
-
2It doesn't have to have a
__len__()method.__getitem__alone with the expected behaviour is sufficient. Jun 27, 2019 at 14:05
If you looking for something short and simple, maybe it will be enough for you:
class A(object):
def __init__(self, l):
self.data = l
def __iter__(self):
return iter(self.data)
example of usage:
In [3]: a = A([2,3,4])
In [4]: [i for i in a]
Out[4]: [2, 3, 4]
Include the following code in your class code.
def __iter__(self):
for x in self.iterable:
yield x
Make sure that you replace self.iterablewith the iterable which you iterate through.
Here's an example code
class someClass:
def __init__(self,list):
self.list = list
def __iter__(self):
for x in self.list:
yield x
var = someClass([1,2,3,4,5])
for num in var:
print(num)
Output
1
2
3
4
5
Note: Since strings are also iterable, they can also be used as an argument for the class
foo = someClass("Python")
for x in foo:
print(x)
Output
P
y
t
h
o
n
All answers on this page are really great for a complex object. But for those containing builtin iterable types as attributes, like str, list, set or dict, or any implementation of collections.Iterable, you can omit certain things in your class.
class Test(object):
def __init__(self, string):
self.string = string
def __iter__(self):
# since your string is already iterable
return (ch for ch in self.string)
# or simply
return self.string.__iter__()
# also
return iter(self.string)
It can be used like:
for x in Test("abcde"):
print(x)
# prints
# a
# b
# c
# d
# e
-
2As you said, the string is already iterable so why the extra generator expression in between instead of just asking the string for the iterator (which the generator expression does internally):
return iter(self.string). Jun 27, 2019 at 14:07 -
@BlackJack You're indeed right. I do not know what persuaded me to write that way. Perhaps I was trying to avoid any confusion in an answer trying to explain the working of iterator syntax in terms of more iterator syntax. Jun 27, 2019 at 16:40
This is an iterable function without yield. It make use of the iter function and a closure which keeps it's state in a mutable (list) in the enclosing scope for python 2.
def count(low, high):
counter = [0]
def tmp():
val = low + counter[0]
if val < high:
counter[0] += 1
return val
return None
return iter(tmp, None)
For Python 3, closure state is kept in an immutable in the enclosing scope and nonlocal is used in local scope to update the state variable.
def count(low, high):
counter = 0
def tmp():
nonlocal counter
val = low + counter
if val < high:
counter += 1
return val
return None
return iter(tmp, None)
Test;
for i in count(1,10):
print(i)
1
2
3
4
5
6
7
8
9
-
1I always appreciate a clever use of two-arg
iter, but just to be clear: This is more complex and less efficient than just using ayieldbased generator function; Python has a ton of interpreter support foryieldbased generator functions that you can't take advantage of here, making this code significantly slower. Up-voted nonetheless. Feb 24, 2018 at 1:30
class uc_iter(): def __init__(self): self.value = 0 def __iter__(self): return self def __next__(self): next_value = self.value self.value += 2 return next_value
Improving previous answer, one of the advantage of using class is that you can add __call__ to return self.value or even next_value.
class uc_iter():
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
next_value = self.value
self.value += 2
return next_value
def __call__(self):
next_value = self.value
self.value += 2
return next_value
c = uc_iter()
print([c() for _ in range(10)])
print([next(c) for _ in range(5)])
# [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
# [20, 22, 24, 26, 28]
Other example of a class based on Python Random that can be both called and iterated could be seen on my implementation here