Minecraft mapping - Reading Minecraft files

(Part of a series on rendering top-down Minecraft maps.)
For storage, the current version of Minecraft breaks the world up into 512×512 cell regions and breaks regions up into 16×16 cell chunks. (These are the X×Z dimensions, regions and chunks are both a full 128 cells tall in the Y dimension.) Previous versions stored each chunk as a separate NBT file, but recent versions bundle up a whole region of chunks into a single MCR file. We can use this NBT parser for Python, but (as far as I can tell) it doesn't know anything about MCR files.
MCR files are really just containers for compressed NBT files. The format is described here: Beta Level Format. We can use that information to seek to the compressed chunk data, decompress it, then feed the decompressed data to the NBT parser.

def read_nbt_from_mcr_file(mcrfile, x, z):
    """
    Read NBT chunk (x,z) from the mcrfile.
    0 <= x < 32
    0 <= z < 32
    """
    #read metadata block
    block = 4*(x+z*32)
    mcrfile.seek(block)
    offset, length = unpack(">IB", "\0"+mcrfile.read(4))
    if offset:
        mcrfile.seek(offset*4096)
        bytecount, compression_type = unpack(
                ">IB", mcrfile.read(5))
        data = mcrfile.read(bytecount-1)
        decompressed = decompress(data)
        nbtfile = NBTFile(buffer=StringIO(decompressed))
        return nbtfile
    else:
        return None

The parsed NBT file gives us access to various pieces of data. This is described here: Chunk file format. We're most interested in "Blocks", the array of block IDs for the blocks in the chunk. Later on we'll also need "Data", because this distinguishes subtle details of certain blocks, such as the colour of wool blocks and the orientation of minecart track blocks. "SkyLight" and "BlockLight" will let us apply lighting to blocks based on illumination from the sky (skylight) and from torches, lava, glowstone and so on (blocklight).
To better manipulate this data, we're going to put it in a numpy array. We use a 3D "structured array" with fields for each of the four pieces of data we've identified. The class VolumeFactory provides various methods that create volumes:

class VolumeFactory(object):
    def empty_volume(self, dimensions):
        data = numpy.zeros(
            dimensions,
            dtype = [
                ('blocks', 'u1'),
                ('data', 'u1'),
                ('skylight', 'u1'),
                ('blocklight', 'u1')])
        return Volume(data)

Empty volume just creates a volume containing 0 for every field. It uses numpy.zeros to create a structured array of the appropriate dimensions.

    def load_chunk(self, nbtfile, volume=None):
        if volume is not None:
            if volume.dimensions != (16,16,128):
                raise TypeError(
                    "load_chunk requires a volume "+
                    "that is 16x16x128")
        if nbtfile is None:
            if volume is None:
                return self.empty_volume((16,16,128))
            volume.blocks[:,:,:]=0
            volume.skylight[:,:,:]=0
            volume.blocklight[:,:,:]=0
            volume.data[:,:,:]=0
            return volume
        if volume is None:
            volume = self.empty_volume((16,16,128))
        level = nbtfile['Level']
        blocks = arrange_8bit(level['Blocks'].value)
        skylight = arrange_4bit(level['SkyLight'].value)
        blocklight = arrange_4bit(level['BlockLight'].value)
        data = arrange_4bit(level['Data'].value)
        volume.blocks[:, :, :] =  blocks
        volume.skylight[:, :, :] = skylight
        volume.blocklight[:, :, :] = blocklight
        volume.data[:, :, :] = data
        return volume

Apart from the handling for non-existent chunks, load_chunk takes an already-loaded NBT file, and uses arrange_4bit and arrange_8bit to assemble that data into the numpy arrays. (Numpy doesn't allow elements crammed together on boundaries smaller than 1 byte, so we pad the 4-bit values out to 8-bits.)

    def load_region(self, fname):
        f = open(fname, "rb")
        region = self.empty_volume((512,512,128))
        chunk = self.empty_volume((16,16,128))
        for z in xrange(32):
            for x in xrange(32):
                chunkdata = read_nbt_from_mcr_file(f, x, z)
                self.load_chunk(chunkdata, volume=chunk)
                region[16*x:16*(x+1), 16*z:16*(z+1), :] = chunk
        return region

Finally, load_region calls load_chunk repeatedly for the 32×32 chunks in a region. It uses numpy's array slicing to paste them together into one big array. Many regions won't be fully-explored, and so many of the chunks will be missing. Those are just filled in with zeroes.
The main program is largely unchanged from last week's zooming and panning one. All that is changed is how we fill the "minecraft_map" texture:

    volume_factory = VolumeFactory()
    region_volume = volume_factory.load_region(
            'world/region/r.0.0.mcr')
    map_rgb_array = numpy.zeros((512,512,3), dtype="u8")
    map_rgb_array[:,:,0] = region_volume.blocks[:,:,70]
    minecraft_map = make_surface(map_rgb_array)

Here I load a minecraft region from a region file I have on my computer called "world/region/r.0.0.mcr". You can find your own region files on your Minecraft saves folder. (On Linux this is in "~/.minecraft/saves", I'm not sure where it ends up on other systems. Perhaps %APPDATA% on Windows.) For now, since we're not yet doing anything smart to flatten the Minecraft map, we just take a slice at y=70. This is just a little bit above sea-level, so it contains a mix of air and land. You can see the results in the annotated screenshot below:

As before, the full code is up on github. The new files are "minecraft_mapping.py" and "nbt_demo.py", the latter of which can be run. You'll need to edit it though, because the path to the Minecraft region file is hard-coded and you'll want to point it towards one of your own map files.
On Friday, we'll look at applying Minecraft's own textures to the map, and next week we'll get on to flattening the 3D map data into something (moderately) useful for a map.