Note to self: explore writing a toy multi-core mapreduce with ‘import multiprocessing’…

You could just use disk directly, so instead of a dict you use a shelve or bsddb, however the problem with that is that you then have a performance hit as all operations are disk based.

You could have a specialization of such a data structure that tries to use RAM however it spills data to disk when necessary.

I’ve been playing with implementing MapReduce in Python, and for the case when the data set doesn’t fit into RAM you need to stream a potentially large, unordered sequence of tuples and then sort them.

The basic use case is that there is any number of Append() calls, and then when you’re done you want the data sorted.

I wrote InfiniteSortedObjectSequence which will be used in my Pythonic MapReduce code that works on large data sets (TBD), however it’s useful in isolation, thus this note.

The code is http://code.google.com/p/tropo/source/browse/trunk/Python/tr_mapreduce/infinite.py. You’ll also need pickle_io and mergesort from that same directory.

The basic way it works is:

infinite = InfiniteSortedObjectSequence()

infinite.Append(...) # call this many times

sorted = infinite.Sort()
for tuple in sorted:
    ...

Behind the scenes when it spills to disk it sorts the “run”, then writes it to a pickled file. There are options to compress the data when it’s written to disk, which unfortunately takes longer but uses less space.

To sort it does an in-memory sort if the data set is small enough, else it does a merge sort using an N-way merge sort.

This seems to work well for a data set that is larger than RAM but which fits on a hard drive. More hard core industry strength solutions would probably not be in python and would try to utilize more disks, control the number of files open during a merge, and maybe use async i/o.

Note that this class is for a specialized use case that is somewhat normal in some aspects of IR and text processing: first you gather the data, then you want it sorted — there’s no random access lookup, length call, etc, just Append() and Sort().

The goal of here is for a simple and short implementation, and with comments stripped out we have:

def MrSimple(producer, mapper, reducer, consumer):
    stage1 = []
    for n, v in producer():
      for n2, v2 in mapper(n, v):
        stage1.append((n2, v2))
    for n2, vals in itertools.groupby(sorted(stage1), lambda x: x[0]):
      seconds = (second[1] for second in vals)
      for v2 in reducer(n2, seconds):
        consumer(n2, v2)

..

producer is a generator that yields a series of name, value pairs – in the classic term frequency counting case it would return file,contents pairs.

mapper takes in name,value pairs and generates a series of name2,value2 pairs. In the word freq case it would emit (term,’1′) pairs for every word in ‘value2′.

reducer is called with (name, values) and emits ‘value3′ that are associated with the name.

consumer is used to persist the results of reducer.

This MapReduce runs in three stages:

1. Run producer and mapper.
2. Sort the name,value pairs the mapper returned.
3. Run the reducer and consumer.

A compile of implementation notes are:

• Ideally I would use the builtin map() however I think that would complicate the code.
• I do use itertools.groupby() which is very handy.

A real implementation would use multiple threads, multiple processes, and be able to process data sets larger than fit into memory.

The core code is in mr_simple.py and a demonstration driver is mr_simple_demo.py. I’ve recently started storing any personal projects that I’m not totally embarrassed by in code.google.com BTW.

To follow – a variation that can work on larger data sets.

For a similar stab at it see this.