Iterators and Generators

1. Iterators and Generators Thomas Wouters XS4ALL thomas@xs4all.net

2. Overview • Iteration • Iterators • Generators • Tasklets • Questions

3. Iteration • The act of going over a collection • Explicit in the form of ‘for’ • Implicit in many forms: • list(), tuple(), dict(), … • map(), reduce(), zip(), … • ‘in’ in absence of __contains__() • ‘extended call’ syntax: func(*…) • but not apply() • Uses __getitem__ in Python before 2.2

4. Iterators • Added in Python 2.2 • Protocol of 2 methods (no special class): • __iter__(): get iterator • next(): get next value • raises StopIteration when done • Explicit iterator creation with iter() • Turn iteration(-state) into objects • Interchangeable with iterable for iteration

5. iter() • Creates a new iterator for an object • Calls __iter__() for creation • Falls back to __getitem__() • Called implicitly for iteration • Wraps a function in an iterator • iter(f, o) calls f until o is returned: for line in iter(file.readline, ""): handle(line)

6. Examples >>> l = [1, 2, 3, 4, 5, 6] >>> it = iter(l) >>> for num in it: ... if num > 1: break >>> for num in it: ... print num; break 3 >>> print list(it) [4, 5, 6]

7. Writing Iterators class IRange: def __init__(self, end): self.end = end self.cur = 0 def next(self): cur = self.cur if cur >= self.end: raise StopIteration self.cur += 1 return cur def __iter__(self): return self

8. Generators • Optional in Python 2.2 • from __future__ import generators • Use new keyword 'yield' • any function with 'yield' is special • Turn function-state into objects • Use the iterator protocol • Not unavoidable • just very very convenient

9. IRange generator >>> def irange(end): ... cur = 0 ... while cur < end: ... yield cur ... cur += 1 >>> print irange(10) <generator object at 0x4701c0> >>> print list(irange(10)) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] • Many useful generators in the 'itertools' module (Python 2.3)

10. Example def map(func, iterable): result = [] for item in iterable: result.append(func(item)) return result def imap(func, iterable): for item in iterable: yield func(item)

11. 'yield' and 'try'/'finally' • Python does not allow 'yield' inside a 'try' block with a 'finally' clause: try: yield x yield x finally: print x • 'yield' inside 'finally' or in 'try'/'except' is allowed

12. Generators as Tasklets • Write function as generator • at all pause points before the final result, yield a sentinel value (such as None) • avoid blocking functions, or write them as generators and call in a loop • Convince callers to use iteration • Remember: you may never be called again

13. TryAgainLater = object() class Bucket: def __init__(self): self.data = [] self.done = False def __iter__(self): return self def next(self): if self.data: return self.data.pop(0) if self.done: raise StopIteration return TryAgainLater def add(self, item): self.data.append(item) def end(self): self.done = True

14. class Tasklet: def __init__(self, *args, **kwargs): self._work = self._workgen(*args, **kwargs) def run(self): try: return self._work.next() except StopIteration: return None def _workgen(self, bucket): for item in bucket: if item is TryAgainLater: yield TryAgainLater continue for returnval in self._process(item): yield returnval def _process(self, item): while not item.done: yield TryAgainLater yield item.value

15. class Tasklet: def __init__(self, *args, **kwargs): self.run = self._workgen(*args, **kwargs).next def _workgen(self, bucket, process): for item in bucket: if item is TryAgainLater: yield TryAgainLater continue for endmarker in process(item): if endmarker is TryAgainLater: yield TryAgainLater break else: raise SuitableError, "process never returned anything" for skipitem in bucket: # eat bucket items until endmarker if skipitem is not endmarker: yield TryAgainLater break else: warnings.warn(SuitableWarning, "marker not found") yield DoneProcessing raise InternalError, "Tasklet finished"

16. class Tasklet: def __init__(self, *args, **kwargs): self.run = self._workgen(*args).next def _workgen(self, bucket, process): for item in bucket: if item is TryAgainLater: yield TryAgainLater continue for endmarker in process(item): if endmarker is TryAgainLater: yield TryAgainLater break else: raise SuitableError

17. # for item in bucket: # ... for skipitem in bucket: # eat bucket items until # endmarker if skipitem is not endmarker: yield TryAgainLater break else: warnings.warn(SuitableWarning, "marker not found") yield DoneProcessing raise TaskletError, \ "Tasklet finished"

18. Questions ?