Well now that the RAID data has been recovered, and a new one is up and running, we can get back to some of the regular programming around these parts. Seems with my ADD and jumping around between a number of different formats, though mostly all from MicroProse, I’ve left some loose ends of unfinished work. I think it’s time we start wrapping up those loose ends, especially with regards to the MicroProse PIC file format. This post is mostly intended as a recap of what we’ve covered so far, to see what is still outstanding.
My plan is to wrap up with each one of the remaining items, and then officially release the source code we’ve developed here so that people can put it to use. I’ve already gone back and retro-actively added links to github repositories with code for some of the other formats already covered and completed.
Looking back at what we’ve covered so far the following things appear to be still open. Some having more remaining to do than others, but for the most part it is just gluing the bits and pieces we have already established together to make a usable utility. At a minimum we have the ability to read and decode all of the formats, but most lack the ability to go back the other way, though we have at least done that work partially for the compressed formats.
MicroProse CAT File Format
As we discovered in my post back when we first looked at the CAT file format, it’s a pretty simple format aggregating several files into one. The file starts off with an index that contains the names, locations, and sizes of the files it contains within. The data itself is a raw copy of the source file with no translation, and can be any kind of content, not just images. Writing a packer for this should be pretty straight forward.
MicroProse SPC File Format
When we looked at the SPC file format we found it to be similar in structure to what we see with the CAT file format, though without any names for the individual files within, and the data is always image data. The image data itself is uncompressed and stored in a planar fashion, with each 32 bit entry holding one byte of data for each of the four planes. The encoder here will be slightly more involved as we have to decompose 4 bit pixel values into 4 separate bit streams and then pack those bit streams into bytes again. The data is stored in individual scanline records, and each scanline has zero based trimming applied to the head and tail, trailing lines of all zeros are also omitted, though prefixed lines are not.
MicroProse PIC File Format
In looking across a large number of DOS/PC based MicroProse titles, we uncovered four main variants of the PIC file format. We have named each of the main variants we’ve found by the year in which they first appeared, based on our findings. With the exception of PIC93, all of the main variants at their core share the same LZW compression scheme. Each variant adding additional meta-data for the image to control its rendering. The main outstanding thing here is to write a core set of core PIC functions to handle the LZW compression and decompression of the image data, as well as the RLE encoding, and decoding of the pixel data. Then wrapping those core functions with the appropriate header and additional meta-data for the particular variant we want.
The MicroProse PIC88 Variant
The PIC88 variant is the oldest, and simplest, variant of the PIC file format we’ve encountered. It consists of a single byte header that declares the maximum number of bits for he LZW compression, and whether the pixel data is packed or not.
The MicroProse PIC89 Variant
The PIC89 variant built upon the PIC88 variant, by adding in some additional header information, namely the image dimensions. It also introduced a number of sub-variants of its own, allowing for different kinds of additional meta-data to be included with the image. We’ve named each sub-variant after the hex value that seems to identify them in the header. For this format we will need to add the ability to store, and recall the additional meta-data as it is not something that can be conveyed in a standard image format. We will need to do the reverse with palette data, as that is stored in a separate file, so we will want to read that and incorporate that into our modern output format, and vice-versa when generating a PIC89 image.
The MicroProse PIC90 Variant
The PIC90 variant is the last of the LZW compressed PIC formats we found, and like PIC89, it builds upon its predecessor. PIC90 is the closest we come to a proper image format where each of the different parts of the file are prefixed with an identifying tag. Rather than having various sub-types like PIC89 to include different meta-data, PIC90 places the different meta-data into its own tagged block. With this format pretty much everything is contained within the image file, but we will still need to be able to save/recall the meta-data that is not capable of being stored in our modern output image format.
The MicroProse PIC93 Variant
The PIC93 variant stands out from the others as it was only ever used with one title that we’ve found so far, and it uses an entirely different compression scheme (LZSS) than it’s predecessors. This is likely due to the fact that the one title it was used with, Railroad Tycoon Deluxe (RRDX), was a port from the Japanese release of the game, which itself was a port/rewrite of the original Railroad Tycoon (RRT) for the Japanese market. In our previous posts for the PIC93 format, we wrote both the LZSS decompressor, and compressor. All that remains is to wrap it up with the appropriate header information, to properly read-write this variant.
So it looks like we have the following 4 main items that need to be tied up.
- MicroProse CAT file format packer
- MicroProse SPC file format packer/encoder
- MicroProse PIC88, PIC89, PIC90 file format decode/encode stack (LZW Compressed Images)
- MicroProse PIC93 file format decode/encode stack (LZSS Compressed images)
Over the next few weeks we’ll finish off on each of the above open items and release the related code. I will likely conclude with the PIC88-PIC90 code, as that is the centrepiece of most of the content so far. With that I think I’ll start from the easiest and work up to the most involved, which happens to be the PIC formats. So next time we’ll kick things off by wrapping up the loose ends with the CAT file format.
Ouch my eye! 
Leave a comment