Batching Pattern

Introduction

The Batching Pattern is used to group things into batches of N items each, and do something with those batches.

Problem

You have I items. You want to put them into N batches of M items each, with one batch containing fewer items than the others. You want to do something with these batches.

Solution

Using Iterators, this is trivial.

while there's more to do:
  start a new batch
  while there's more to do and the batch isn't full:
    pull an item off and put it in this batch.
  process this batch

or

while there's more to do:
  pull off up to M items
  process them

As a function this would take the following parameters:

• Iterable of items to batch
• Size of each batch
• A function to process a batch that takes an iterable of items.

This could be implemented in Python as:

from itertools import islice
def batch(iterable, size, process_func):
    it = iter(iterable)
    try:
        while True:
            process_func(islice(it, size))
    except StopIteration:
        pass