Today’s format has been on my list since the early days of my journey in this reverse engineering of image file formats started about a year ago. My interest in it was re-kindled by a recent comment by “RedMike” to my PIC File Format summary. The format in question is the .PAN File format from MicroProse. PAN files are animations, so I’m going to assume PAN stands for PIC Animation. Lets see if we can help Mike out and help to figure this one out.
From Mike’s comment, and further conversations, he is trying to make an editor for “Covert Action” (1990). Like me he has the itch to not leave something untouched/unknown even if it isn’t strictly necessary. Now Mike has been chipping away at this for a little while and has a head start on me with findings, so I will need to rely on him for a lot of things until I can get caught up. Hopefully I can get there pretty quick, and start to contribute to the effort. With that said, lets dig into it.
First Look
As we always do, we start by pulling up one of the target files in a hex editor to see what we’re looking at. I usually try to grab a file that would be used early in the game, as then it’s easier to see it when it comes to testing later. In this case I’m going with TITLE2.PAN which should be from the opening animation if I’m not mistaken.
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000 50 41 4E 49 03 01 01 00 03 01 02 03 04 00 06 07 PANI............
00000010 08 09 0A 0B 0C 0D 0E 0F 00 00 00 00 00 3F 01 C7 .............?..
00000020 00 01 00 02 01 00 9B 77 07 78 3E 78 69 78 97 78 .......w.x>xix.x
00000030 C2 78 F4 78 00 00 F8 78 FB 79 01 7B 4D 7B 78 7B .x.x...x.y.{M{x{
00000040 A8 7B 00 00 00 00 00 00 00 00 00 00 00 00 E8 7B .{.............{
00000050 EB 7B 00 00 16 7C 00 00 00 00 00 00 00 00 CC 7C .{...|.........|
00000060 00 00 23 7D 8F 7D C6 7D F1 7D 1F 7E 4A 7E 00 00 ..#}.}.}.}.~J~..
00000070 00 00 00 00 00 00 7C 7E C8 7E F3 7E 23 7F 00 00 ......|~.~.~#...
50 41 4E 49: "PANI" Format Tag?
3F 01: 0x013f (319) Width-1? or perhaps right side limit?
00 C7: 0x00c7 (199) Height-1? or perhaps bottom limit?
00 00 00 00: lower limits/bounds?The first thing that jumps out at me is that this does not look familiar at all, and I KNOW that I had peeked at a PAN file before. What I recall seeing previously was pretty much a binary blob, with no immediately identifiable parts. Here we can clearly see a “PANI” as an identifying tag. Furthermore we can see some values that look to be dimensions, though one less than what we would expect (319×199) instead of (320×200). With that said, there is a large enough sequence of 00‘s before that so perhaps these are TopLeft and LowerRight bounds for the image, making it a 320x200 image, which makes sense. But now I’m puzzled, what did I look at before? After a little head-scratching, I remembered it was with F117 where I looked at a PAN file. Looking at TITLE-1.PAN from F117 we see the following:
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000 FF FE 5A 14 6F 00 C3 4D 07 00 FE F8 23 36 10 9B ..Z.o..M....#6..
00000010 10 25 CA 11 12 D4 F8 25 FE F8 89 FF 2A FE 80 A7 .%.....%....*...
00000020 FD FF F6 2A 15 F5 14 01 15 FF 3F FE FD 18 F1 FF ...*......?.....
00000030 F6 F9 FC 18 25 FF FF 25 18 24 24 18 23 23 18 22 ....%..%.$$.##."
00000040 22 18 21 21 18 20 20 FF FF 18 1F 1F 17 1E 1E 17 ".!!. .........
00000050 1D 1D 16 1C 1C 15 1B 1B 14 FF 3F 1A 1A 13 19 19 ..........?.....
00000060 12 18 18 11 17 17 10 16 16 0F FC FF C4 0E 14 14 ................
00000070 0D 13 13 0C 12 12 0B 11 11 0A FF FF 10 10 09 0F ................Now that’s more like what I remember seeing. Though looking at it now, I think I see a bit of an identifiable pattern, but I think we’ll save that for another post. The task at hand is the version we see with “Covert Action“, so we can help Mike out. This does tell us though that there are at least two variants of this format. (As I was going through this format, I did encounter a 3rd variant, more similar to this one which I will mention later as it does help guide some choices going forward)
What do we already know?
Before we dig any deeper, what is it we already know about the format? Mike mentioned that the format seems to contain PICv2 formatted images. Indeed if we look a bit lower in TITLE2 we do see what looks to be a PICv2 header pattern.
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000210 00 00 00 00 00 00 00 00 00 00 07 00 38 00 35 00 ............8.5.
00000220 0B 55 20 31 F8 07 09 41 40 03 F5 FE 05 54 40 90 .U 1...A@....T@.
00000230 C1 97 2A 10 FF 29 2C 98 10 D2 83 2F 09 27 1E A8 ..*..),..../.'..
07 00: PICv2 Type 7 (4-bit packed)
38 00: Width 0x0038 (56)
35 00: Height 0x0035 (53)
0B: LZW max-code size (11 bits)That seems to confirm that PAN may very well stand for PIC Animation. Also from the dimension this would suggest that the image seems to be broken into smaller sub-images. Using a search to look for the pattern 07 00 I find only 47 matches, and some of these are likely to be false matches for a full PICv2 header. But at most we would have fewer than 50 image fragments here. Far to few I think to be individual frames. That suggests to me we are not looking at some sort of delta encoding, as we see in video files (back then several “animation” formats were more like a primitive video format), but rather perhaps a true sprite animation format. In this case we would expect to see some number of sprite images, and then some sort of control sequence that tells the animation-player how to move the sprites around. Extending my search to look for “07 00 ?? ?? ?? ?? 0b” to look for possible images allowing for varying image dimensions, but a constant 11 bit LZW setting, I reduce the original 47 matches down to 37. Further if we look at the instances of the matches we can see varying image dimensions, further supporting the sprite hypothesis.
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000210 07 00 38 00 35 00 ..8.5.
00000220 0B .
07 00: PICv2 Type 7 (4-bit packed)
38 00: Width 0x0038 (56)
35 00: Height 0x0035 (53)
0B: LZW max-code size (11 bits)
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000310 07 00 27 00 26 00 0B ..'.&..
07 00: PICv2 Type 7 (4-bit packed)
27 00: Width 0x0027 (39)
26 00: Height 0x0026 (38)
0B: LZW max-code size (11 bits)
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
000003C0 07 00 2A 00 25 00 0B ..*.%..
07 00: PICv2 Type 7 (4-bit packed)
2A 00: Width 0x002a (42)
25 00: Height 0x0025 (37)
0B: LZW max-code size (11 bits)One other thing I noticed when searching, all of the PIC headers I found also always seem to start on an even address boundary, but appear to be backed one after another in the file. This leads me to think that there may be some padding added after the image data to enforce word alignment, as it is unlikely that the LZW compressed data always falls on an even boundary when done. Despite what we see in the header section of the file for some values, it appears the file relies on word alignment to some extent.
Beyond what Mike mentioned in his comments, and our discussions, a quick search dug up this Git repository from Jari Komppa that seems to have a little more info. I actually stumbled across Jari’s repository back when I was looking an the CAT file format, but never picked up on the PAN file detail he had there. Now the info there is far from complete, but may serve as a bootstrap to help get us going. Here is an excerpt from his formats.txt with respect to the PAN format.
11. PAN
-------
14560 BUSTOUT.PAN
11102 RESEARCH.PAN
8838 WARNING.PAN
8232 ESCAPE.PAN
15840 BUGCAR.PAN
16268 SURNDR01.PAN
14924 OFFICEM.PAN
9952 CRYPTO.PAN
18816 BRIEFING.PAN
19418 BLDING04.PAN
7718 FOLLOWED.PAN
15750 BLDING03.PAN
7476 CAPTRD01.PAN
15030 OFFICEF.PAN
16364 BLDING02.PAN
9846 TITLE2.PAN
11180 COMM.PAN
7866 HQ.PAN
16920 BLDING01.PAN
16680 INJAIL.PAN
12228 BINOS3.PAN
11402 INTRGATE.PAN
16962 CREDITS.PAN
21710 WOUNDED.PAN
PAN files are the graphics for all of the animated sequences, starting
with a PANI tag which suggests they're related to some tool not
specific for this game (as game data files are tagless).
All of the files appear to contain the same first 28 bytes, most
likely containing stuff like resolution. The data itself is very
likely to be compressed.
The format doesn't seem to be FLI/FLC (autodesk) or ANM (deluxe paint
animator) which were popular at the time.. however, some sort of RLE
scheme is likely to be in use, so that only the changed parts of the
image are stored. Note how the WOUNDED animation is the largest file;
the animation changes a lot of screen pixels every frame.
Header
ofs | datatype | description
----+----------+------------
0 | 32 bit | header tag 'PANI'
4 | byte | always 03
5 | byte | always 01
6 | byte | always 01
7 | byte | always 00
8 | byte | always 03
9 | 15 bytes | color mapping table from color 1 onwards
| | (color 5 maps to color 0)
24 | 5 bytes | always 0
29 | 16 bit | width -1
31 | 16 bit | height -1
33 | 16 bit | ? 1, 3, 4, 5
35 | byte | ? 0, 1, 2,
36 | 16 bit | ? 0, 1, 7, 5
38 | 16 bit | ? frame count?
40 | 16 bit | ? often 0xc8
42 | 8 bit | ? often 0x0b
In 18 of the animation files, byte at offset 42 is 0xb, followed by
a variety of different kind of data, hinting that the animation
frames might be delta-encoded PIC data frames.
When offset 42 is 0xb, offset 35 is always 1 and offset 36 is
always 7, so these may be format flags. For all other files
in this category, offset 33 is always 1, except for bustout.pan,
for which it is 3.
If offset 40 is 0xc8, offset 42 is always 0xb. binos3.pan has
0xb at offset 42, but offset 40 isn't 0xc8.
For files where offset 42 isn't 0xb, a 0xb byte can be found
later on (offset 0x220 for blding*.pan and title2.pan, but
0x21e for credits.pan).While Jari seems to have made the connection with PIC here, he seems to believe this is more like a video format with delta encoding than a series of sprites. The dimensions, and quantity of individual images suggest sprites rather than frames. Regardless, We do have a bit of a starting point here, that also largely aligns with my first impressions of the data when I looked at it above.
Applying what we know
So to kick things off, lets see if we can apply what we know to see how that results in parsing of the file, and to see if we can’t come up with a more concise definition. (disclosure here, this is a bit of a condensed version, with some discoveries/realizations being presented out-of order to when they really happened to avoid too much confusion. The process was a messy path)
The first thing we encounter with the file is the obvious tag “PANI“, then a number of unknown bytes. Now in Jari’s notes there are 5 bytes, though I suspect it’s only 4, as it makes sense to end on an even boundary. I’ll get the the remaining byte next. That leaves us with 8 bytes. The first 4 being a fixed tag, and the next 4 possibly being a version number, and/or perhaps capabilities flags. I’m going to assume this is a version value for now, and will consider these combined to be the files identifier. (we’ll see a little later that version does make sense here, at least for the first byte)
typedef struct {
char tag[4]; // "PANI"
uint8_t version[4]; // 03 01 01 00
} pan_id_t;
Following the header we have the colour map where the 16 colour palette is remapped, effectively forming the palette, so I will consider it as such. Now in Jari’s notes the map is 15 entries, since colour 0 is always transparent. I think that actually we have 16 entries here, with that one remaining byte from earlier being the remapped value for colour 0. It’s just we can never see the effects of that remapping since it is the transparent colour. In testing this value can be safely changed, and no visual effects were observed in-game. Changing any of the other values in the palette does make the expected visual changes. 16 entries makes much more sens to me, and it doesn’t make much sense to prepend a padding byte here. Sadly the implementation here used just the 0-15 range of values to remap the colours, instead of allowing of setting the colours based on the EGA‘s 64 colour palette. Furthermore there is a byte that follows the colour map/palette. I think that may actually be the index value for the colour to make transparent. I wasn’t able to confirm this as changing the value after the colour map also had no visible effects. Perhaps the game engine hard-codes this to 0 as part of an optimization, as checking a value for 0 is faster than checking against some arbitrary value, and you would need to do this for every pixel each time you drew. Padding after this table makes no sense, as this byte actually forces misalignment of anything that follows. (we’ll see later that this group of bytes makes sense to be grouped as the palette)
typedef struct {
uint8_t cmap[16]; // entry 0 has no effect (transparent colour)
uint8_t transparent; // Always 0 - no effect, hard-coded
} pan_pal_t;
After the palette block, we have what looks to be the dimensions for the display size of the animation. We have a pair of values that look to be width and height, but one less, making it more like max_x and max_y. Preceding that we have enough bytes to hold 2 more 16 bit values, possibly representing min_x and min_y. Once again in testing changing the 0 values to anything else had no effect, so these bytes appear to be ignored, even in the size calculation. Though their placement does strongly support their intended use being for the viewport minimum. This may have once again been optimized away for the game. After the two sets of values, we have another 3 bytes of data that vary from file to file. Which brings us to the end of the common structure we see in every file. I’m going to leave those 3 bytes off for the time being, leaving us only with the dimension definitions to form the info block.
typedef struct {
uint16_t x;
uint16_t y;
} point16_t;
typedef struct {
point16_t min; // always 0, change has no effect
point16_t max;
} pan_info_t;
That finishes off on all the common parts we see across all the PAN files we see with “Covert Action“. I’ve made a number of educated guesses on what some of the values are, and in most of those cases altering the values has no effect. As I’ve already mentioned I believe this to be an “intent vs implementation” discrepancy, where some intended functionality was removed or hard-coded in the name of optimization for the game engine. Combining all of the above we get our header for all the PAN files we’ve seen so far.
typedef struct {
pan_id_t id;
pan_pal_t pal;
pan_info_t info;
} pan_hdr_t;
There are a few more bytes that come after the header, that while their values change from file to file, the range of values are small, and the structure of them remains consistent. Jari touches on these bytes in his description a bit, and we will tackle them shortly. Beyond that Mike did mention a 502 byte block of data. I can see that, but its location seems to move depending on the file, so we will set that aside for the moment. We also know that the sprites look to be PICv2 encoded, and align on word boundaries. With that I think we have enough that we can start parsing the file. We have a few details to work out still after the header, but once we get into the images, I think we have smooth sailing until the end of the file, where once again we appear to have a block of unknown data.
Additional Reference Data
While working on figuring out the structure for the PAN files, I searched through the assets for my various MicroProse titles, and found 2 more titles that had PAN files that contained the “PANI” marker we see here. The first find was with the original version of “Railroad Tycoon“. With RRT everything seems to match what we see here with “Covert Action” with the exception that the value on the zero position of the colour map differs with some of the PAN files there. This is what made me lean towards this being part of the colour map, as there I saw a value of 5, effectively making a swap between the normal black and magenta entries in the palette.
The second place I found PAN files with the “PANI” was with “LightSpeed” (I believe “HyperSpeed“ will also be the same, as it was developed at the same time with the same engine). However this time the underlying PIC encoding is that of PICv3 instead of PICv2 that we see with “Covert Action” and “Railroad Tycoon“. We also see that the first byte in 4 bytes after the “PANI” tag, changes to a 4 from 3, making me think this may be a version indicator. With this version we see a set of “M0” and “E0” palette and dither map definition blocks in place of the palette/colour map block we have in the version here. Immediately following the palette data is the info block with the dimensions, which is why I grouped things as I did. (I only took a cursory look, so there may be further things that help clarify and define what we have here, but that will be left for a future post)
Locating the table
I think our next challenge is going to be in locating this 502 byte table mentioned by Mike in his comments. Scanning through the files it seems to be in different places, but clearly it is always there. I’ve been using TITLE2.PAN as my Guinea pig as it is the animation that plays when the game is loaded, so when I poke and alter bits, I can quickly see the effects of the changes.
From TITLE2 we see the following sequence immediately after the header. We have 3 bytes of data first, that look to be a 16 bit value, followed by an 8 bit value. That is then followed by what looks to be 251 16 bit values (502 bytes), and then finally we see a PIC image header.
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000020 01 00 02 01 00 9B 77 07 78 3E 78 69 78 97 78 ......w.x>xix.x
00000030 C2 78 F4 78 00 00 F8 78 FB 79 01 7B 4D 7B 78 7B .x.x...x.y.{M{x{
00000040 A8 7B 00 00 00 00 00 00 00 00 00 00 00 00 E8 7B .{.............{
00000050 EB 7B 00 00 16 7C 00 00 00 00 00 00 00 00 CC 7C .{...|.........|
⋮ ⋮
000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000210 00 00 00 00 00 00 00 00 00 00 07 00 38 00 35 00 ............8.5.
00000220 0B .
01 00: Unknown 16 bit value (From Jari: only see 1, 3, 4, or 5 across the files)
02: Unknown 8 bit value (From Jari: only see 0, 1, or 2 across the files)
01 00 9B 77 … 00 00 00 00: 502 bytes of data (looks to be 251 16 bit values)
07 00 38 00 35 00 0B: PICv2 header for first imageThe table appears to be very sparsely populated, but the values appear to be ever increasing in value, which leads me to think these might be some sort of offset, perhaps pointing to each of the sprite images. However, they do not appear to be file offsets, as they do not land on, or near, any of the sprite headers we can find in the file. The values also don’t make sense as sizes, so they cannot be the compressed image sizes either. I also checked the difference in the values to see if that could be related to size (and therefore offset) and at first it did seem to work (not exactly, but pretty close), but quickly fell apart after only a couple of images. The deltas seem to be all over the place, and it was just a coincidence that it worked at first. Not sure what to make of these values just yet, let’s look at a few more files to see if any more of a pattern emerges.
For CREDITS.PAN we see only 500 bytes (250 16 bit words) if we are to assume the same 3 byte structure at the top. Which I think we have to do, as there is no obvious mechanism to distinguish a change in the structure before this point. We don’t see a difference in the stream of bytes until we get to the 3rd byte which is a 00 instead of a 02 that we saw before.
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000020 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...............
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000050 00 00 00 00 00 00 D2 52 DE 52 00 00 00 00 00 00 .......R.R......
⋮ ⋮
000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000210 00 00 00 00 00 00 00 00 07 00 40 01 01 00 0B ..........@....
01 00: Unknown 16 bit value (From Jari: only see 1, 3, 4, or 5 across the files)
00: Unknown 8 bit value (From Jari: only see 0, 1, or 2 across the files)
00 00 00 00 … 00 00 00 00: 500 bytes of data (looks to be 250 16 bit values)
07 00 40 01 01 00 0B: PICv2 header for first imagePerhaps that 3rd bye is indicating the presence of extra bytes? We are going from 0 to 2 after all, and we do see two extra bytes when it’s a 2. According to Jari’s notes that third byte also has a value of 1 sometimes. Let’s look at a file which has a 1 there.
The first file I found with a 1 at the 3rd position is BRIEFING.PAN. And here we have an extra wrinkle now we see a PIC image immediately after that 3rd byte and before the table. It’s a bit hard to see the boundary of where the PIC data ends and where the table begins. However When we look at the values for the two words occurring at the 500 and 502 positions, the one at 502 doesn’t really make sense, the value is very different in magnitude than the rest of the values. Going back to TITLE2, we see the same thing, that first value, if the table is 502 bytes, is very different from the rest. So I think it’s safe to say that the table is a fixed 500 bytes. The extra 2 bytes in the case of TITLE2 must be something else. A size of 501 doesn’t make sense here, as it throws off the 16 bit alignment of the values in the table, and we would end up with an odd-length, meaning we have a half entry left-over.
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000020 01 00 01 07 00 40 01 C8 00 0B 33 70 D4 C0 31 .....@....3p..1
00000030 63 94 C1 29 55 8A 50 21 C6 84 08 24 03 4C 98 31 c..)U.P!...$.L.1
00000040 AB 52 63 60 C1 19 15 71 8C 9A 81 91 A3 41 81 38 .Rc`...q.....A.8
00000050 08 6E C4 82 E8 D6 2E 48 1F FE 41 22 F0 EB DF AF .n.....H..A"....
⋮ ⋮
00002120 56 B3 B2 55 AD 70 4D AB 5C DF 3A 57 B7 DA B5 AD V..U.pM.\.:W....
00002130 78 8D 2B 5D F7 7A 57 BD F6 B5 AE 79 05 AC 5F 03 x.+].zW....y.._.
00002140 CB 57 C2 FE B5 B0 83 4D AC 60 01 CB 55 23 D2 56 .W.....M.`..U#.V
00002150 DA 56 E7 56 F8 56 0E 57 1C 57 2E 57 3F 57 4D 57 .V.V.V.W.W.W?WMW
00002160 63 57 79 57 A9 57 D9 57 00 00 E0 57 00 00 00 00 cWyW.W.W...W....
00002170 00 00 00 00 00 00 E8 57 34 58 85 58 D4 58 2B 59 .......W4X.X.X+Y
00002180 7F 59 C9 59 0E 5A 00 00 00 00 5F 5A AF 5A FE 5A .Y.Y.Z...._Z.Z.Z
00002190 49 5B 00 00 00 00 00 00 00 00 00 00 00 00 00 00 I[..............
000021A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
⋮ ⋮
00002300 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00002310 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00002320 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00002330 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00002340 00 00 07 00 18 00 14 00 0B .........
01 00: Unknown 16 bit value (From Jari: only see 1, 3, 4, or 5 across the files)
01: Unknown 8 bit value (From Jari: only see 0, 1, or 2 across the files)
07 00 40 01 C8 00 0B: PICv2 header for first image
55 23: Word at 502 bytes from end of table
D2 56: Word at 500 bytes from end of table
07 00 18 00 14 00 0B: PICv2 header for second imageLets look at one more file with a 1 in that 3rd position. With CAPTRD01.PAN we again have the PIC image before the table, but this time the boundary looks to be much clearer, and it happens at the 500 byte mark from the end of the table. So I think it’s safe to say the table is always 500 bytes in size, and not 502.
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000020 01 00 01 07 00 60 00 C8 00 0B 5F CA 98 81 F4 .....`...._....
00000030 C0 4A 0D 3F 90 36 78 A9 17 70 A0 03 83 E7 20 71 .J.?.6x..p.... q
00000040 B8 C2 50 20 24 07 07 21 75 58 D8 10 52 03 2B E3 ..P $..!uX..R.+.
00000050 22 76 A0 D8 B1 41 46 0F 1C 2D 32 00 19 D1 03 49 "v...AF..-2....I
⋮ ⋮
00000BC0 81 85 29 45 F4 65 7B 27 81 E5 E1 38 6F 4A E8 96 ..)E.e{'...8oJ..
00000BD0 00 67 40 2A D8 C5 82 C8 24 0B 3C B2 C9 25 F7 76 .g@*....$.<..%.v
00000BE0 B2 CA 25 03 00 00 00 00 EF 59 00 00 F0 5A 00 00 ..%......Y...Z..
00000BF0 CF 5B 00 00 AB 5C 11 5D 00 00 00 00 D4 5D 00 00 .[...\.].....]..
⋮ ⋮
00000DA0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000DB0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000DC0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000DD0 00 00 00 00 00 00 00 00 07 00 64 00 49 00 0B ..........d.I..
01 00: Unknown 16 bit value (From Jari: only see 1, 3, 4, or 5 across the files)
01: Unknown 8 bit value (From Jari: only see 0, 1, or 2 across the files)
07 00 60 00 C8 00 0B: PICv2 header for first image
5F CA … 25 03: PIC image data
00 00 00 00 … 00 00 00 00: 500 bytes of data (looks to be 250 16 bit values)
07 00 64 00 49 00 0B: PICv2 header for second imageZooming out
So now we know that the table is 500 bytes or 250 words. Doing a quick count of non zero entries in the table, it seems to match the number of PIC sprites we have in the file. With one exception, when there is a PIC image before the table, we have one extra sprite than entries in the table. So maybe the table only applies to the images that come after. It also makes sense since that image that comes before is most certainly the background image, as it is always the full size as defined in the PAN header. Whereas all the sprites that come after are generally smaller. So the table appears to be some sort of map to the sprites that follow. Mike did mention in his comment that he was having trouble mapping the image ID in the control data to the actual sprites. Perhaps this table is the cross reference that does that. Mike also was able to test, and confirm, that the actual value in this table does not seem to be important, only that it be non-zero. That’s an interesting find, and does make things a bit easier for now, we don’t need to worry about what the value is, only that it’s non-zero. My hypothesis is that that the sprites that come after the table are applied in order to the non-zero entries in the table. Meaning that the index in the table for a given entry is the ID of the sprite for use later. So if the first non-zero entry index is 4, then the first sprite image that follows is referenced in the control code with an ID of 4. (We will be able to confirm this later when we get into the control data)
Now if only we could easily determine when and where the table appears in the data. It was while I was trying to figure this out and was chatting with Mike, I suggested that maybe some flag bits or something in the still unknown bytes of the header is signalling this. That’s when Mike made the correlation between that 3rd byte that comes after the header, and what we see in terms of background. When it is a 1, we always have a bitmap, when it is a 0, there is no background (as seen with CREDITS.PAN, which is played following TITLE2.PAN but appears to keep what TITLE2 left on-screen. When the value is a 2, the background is filled with a single colour. I can’t believe I missed the correlation, it is so obvious now. That likely means the extra 2 bytes we see with TITLE2 before the sprite cross-reference table are defining the fill colour.
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000020 01 00 02 01 00 .....
01 00: Unknown 16 bit value (From Jari: only see 1, 3, 4, or 5 across the files)
02: Background Type "Fill"
01: Background Fill Colour 0x01 (1) ==> blue?
00: Padding or some other colour definition?The startup animation does have a blue background, so this seems to fit. Changing the value from 1 to 2 does indeed change the colour in the animation from blue to green. Changing the 2nd byte from 0 to anything seems to have no effect, so this is likely a padding byte to maintain 16 bit alignment. That now gives us the following structure for the fill.
typedef struct { // used for the background colour, if set
uint8_t colour;
uint8_t zero; // padding, always 0
} pan_colour_t;
Since we also now have a finite set of values for the background, let’s make an enumeration for that. This will make things a bit more readable later.
typedef enum {
background_none = 0,
background_image = 1,
background_fill = 2
} background_type_t;
And with that we need to update our header structure a little too. Note that we don’t use the enum directly here, as then the compiler would likely place an int here and we only want an 8 bit value so we force it. (the actual type used is not defined by the C standard, and is left to the compiler implementation, though int is the most common I’ve seen)
typedef struct {
pan_id_t id;
pan_pal_t pal;
pan_info_t info;
uint16_t unknown; // Only see 1, 3, 4, or 5 here, usually 1
uint8_t background_type; // background_type_t enum values
} pan_hdr_t;
Unfortunately we can’t express what follows, in the file representation, directly as a structure in C. The problem is we have optional and variable length data. Effectively this is a union between nothing, a PIC image, and a background colour. After that we have our cross reference table of 250 entries. This seems to be a fixed maximum, but perhaps there is some value in the header that can increase this size. All the files we’ve looked at however are 250 bytes, so it would be hard to verify this. Also, for some reason the actual values are not important, we can simply treat them as true/false values. If true the index corresponds to a sprite that follows the table. Not sure why MicroProse chose to do it this way in the first place. This table could have easily been a table of file offsets to the respective sprites in the file. Regardless we have the following for the XREF table. We don’t really need to keep this data, as it contains no useful data after the file has been read in. We just need to keep it long enough to map the sprites that follow properly to their ID values.
uint16_t sprite_xref[250]; // only used during reading
While we can’t represent the data in the file directly as a struct in C, we can, easily express it in our memory representation. To make the sprite cross-reference more useful, I chose to convert it to a table of pointers to the sprites, with the sprites ID being the index into the table of pointers. This should make mapping back to the actual sprite easier when we get into the control data.
typedef struct { // memory representation of the PAN data
pan_hdr_t hdr;
pic_v2_t *bkgnd_image; // only valid when hdr.background_type = 1
pan_colour_t bkgnd_fill; // only valid when hdr.background_type = 2
pic_v2_t *sprite[250];
uint8_t control_data[];// remaining data in file
} pan_file_t;
First pass
At this point I think we have everything we need to take a first pass at parsing the file in its entirety. We still don’t know what the control data is, but it appears to always be at the end of the file, so we can safely assume that everything that remains after the last sprite is control data. So I wrote up some code to parse through the file. In this case only calculating the sizes of the decompressed images, and not extracting anything to files, though that will be a simple change. With that said, let’s see what we get.
MicroProse PAN File Explorer (v.0.1)
Inspecting: 'TITLE2.PAN' Size: 9846 bytes
000000: Header: Version 3 [Extra: 01 01 00]
000008: cmap: [3 1 2 3 4 0 6 7 8 9 10 11 12 13 14 15]
000018: Transparent Colour: 0
000019: Dimensions: 320x200 [(0,0) - (319,199)]
000021: Unknown: [01 00]
000023: Background Type: Fill (Colour: 1, Pad: 00)
000026: Sprite Map: 37 sprites
00021a: Sprite000: 56x53 max-bits: 11 ( 238 bytes) [Table: 779b] [Pad: 90]
000310: Sprite001: 39x38 max-bits: 11 ( 176 bytes) [Table: 7807] [Pad: ff]
0003c8: Sprite002: 42x37 max-bits: 11 ( 173 bytes) [Table: 783e]
00047c: Sprite003: 34x38 max-bits: 11 ( 158 bytes) [Table: 7869] [Pad: 0a]
000522: Sprite004: 35x37 max-bits: 11 ( 126 bytes) [Table: 7897] [Pad: 1d]
0005a8: Sprite005: 28x49 max-bits: 11 ( 143 bytes) [Table: 78c2]
00063e: Sprite006: 9x3 max-bits: 11 ( 14 bytes) [Table: 78f4] [Pad: ae]
000654: Sprite008: 136x199 max-bits: 11 ( 587 bytes) [Table: 78f8]
0008a6: Sprite009: 141x200 max-bits: 11 ( 533 bytes) [Table: 79fb]
000ac2: Sprite010: 61x53 max-bits: 11 ( 257 bytes) [Table: 7b01]
000bca: Sprite011: 35x38 max-bits: 11 ( 165 bytes) [Table: 7b4d]
000c76: Sprite012: 21x49 max-bits: 11 ( 115 bytes) [Table: 7b78]
000cf0: Sprite013: 44x37 max-bits: 11 ( 161 bytes) [Table: 7ba8]
000d98: Sprite020: 30x1 max-bits: 11 ( 4 bytes) [Table: 7be8] [Pad: fe]
000da4: Sprite021: 95x13 max-bits: 11 ( 216 bytes) [Table: 7beb] [Pad: 01]
000e84: Sprite023: 30x125 max-bits: 11 ( 13 bytes) [Table: 7c16]
000e98: Sprite028: 219x20 max-bits: 11 ( 538 bytes) [Table: 7ccc] [Pad: 22]
0010ba: Sprite030: 56x53 max-bits: 11 ( 238 bytes) [Table: 7d23] [Pad: 1b]
0011b0: Sprite031: 39x38 max-bits: 11 ( 173 bytes) [Table: 7d8f]
001264: Sprite032: 42x37 max-bits: 11 ( 170 bytes) [Table: 7dc6] [Pad: 2c]
001316: Sprite033: 34x38 max-bits: 11 ( 156 bytes) [Table: 7df1] [Pad: 1d]
0013ba: Sprite034: 35x37 max-bits: 11 ( 125 bytes) [Table: 7e1f]
00143e: Sprite035: 28x49 max-bits: 11 ( 143 bytes) [Table: 7e4a]
0014d4: Sprite040: 61x53 max-bits: 11 ( 251 bytes) [Table: 7e7c]
0015d6: Sprite041: 35x38 max-bits: 11 ( 164 bytes) [Table: 7ec8] [Pad: 81]
001682: Sprite042: 21x49 max-bits: 11 ( 113 bytes) [Table: 7ef3]
0016fa: Sprite043: 44x37 max-bits: 11 ( 161 bytes) [Table: 7f23]
0017a2: Sprite050: 56x53 max-bits: 11 ( 238 bytes) [Table: 7f63] [Pad: 10]
001898: Sprite051: 39x38 max-bits: 11 ( 175 bytes) [Table: 7fcf]
00194e: Sprite052: 42x37 max-bits: 11 ( 173 bytes) [Table: 8006]
001a02: Sprite053: 34x38 max-bits: 11 ( 155 bytes) [Table: 8031]
001aa4: Sprite054: 35x37 max-bits: 11 ( 126 bytes) [Table: 805f] [Pad: 75]
001b2a: Sprite055: 28x49 max-bits: 11 ( 143 bytes) [Table: 808a]
001bc0: Sprite060: 61x53 max-bits: 11 ( 255 bytes) [Table: 80bc]
001cc6: Sprite061: 35x38 max-bits: 11 ( 165 bytes) [Table: 8108]
001d72: Sprite062: 21x49 max-bits: 11 ( 115 bytes) [Table: 8133]
001dec: Sprite063: 44x37 max-bits: 11 ( 161 bytes) [Table: 8163]
001e94: Remaining data: 2018 bytes
002676: File Parsing completeWell I’d call that a success. We made it to the end of the file without any breaks. I ran this against a few of the other files, and we successfully parse those as well. Last major hurdle is the control data, so let’s take a first look at that now.
Hidden figures
One thing that came up in the decoding here, that we hadn’t seen before with PIC, is how to handle packed images with odd widths. While I handled this in my code, I never really talked about it in my blog post. It didn’t seem all that important at the time, as all the images we were working with were full-screen images, which always landed on an even pixel boundary. With the images here, we are not always landing on an even width, and the images are packed. While one could simply start shifting in pixels from the next line of data, this does not make much sense, as then to decode a pixel from the packed array, one would need to know what line it came from to decide which nibble the pixel belongs to. Also while we are unpacking the images, it is quite likely that the game stopped at the RLE stage, as the packed format is effectively the native format for the video card. Not to mention that memory was quite precious back then. so then how do we handle an odd length line? We simply pad the line out with an extra pixel’s worth of data when encoding. The net effect is that in the packed stream, the image appears to be 1 pixel wider than it really is, if it has an odd width. As such when decoding we need to track where we are horizontally, and ignore that last nibble of data if it would cause us to exceed the specified width. Generally the value in this padded pixel will be 0 or whatever the background colour is, so relatively benign. You probably could get away with decoding it by increasing the width by 1 whenever an odd width is encountered. I wouldn’t count on this not having a negative impact though, as there is no guarantee this won’t create an unwanted line at the right edge of the image.
Control Data First Look
At this stage Mike is still way ahead of me, he’s well into parsing the control data, though is having troubles reliably parsing it as the structure doesn’t always make sense. From his comments, and our discussions the values in the control data seem to be prefixed with 05 00. He also said this only seems to hold for the first part of the control data.
Hex View 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00001E90 7E 00 05 00 B2 06 05 00 01 00 05 00 ~...........
00001EA0 FF FF 05 00 9B FF 05 00 0C 00 05 00 FF 00 05 00 ................
00001EB0 00 00 00 05 00 CA 06 05 00 02 00 05 00 FF FF 05 ................
00001EC0 00 9E FF 05 00 E1 FF 05 00 FF 00 05 00 00 00 00 ................
00001ED0 05 00 E2 06 05 00 03 00 05 00 FF FF 05 00 9C FF ................
⋮ ⋮
00002630 05 1E 00 00 FF 06 9D 07 05 24 00 02 F6 FF 00 00 .........$......
00002640 00 09 06 A5 07 00 09 0A 05 30 00 00 FF 06 B5 07 .........0......
00002650 05 18 00 02 F6 FF 00 00 00 09 06 BD 07 00 09 0A ................
00002660 05 47 00 00 FF 06 CD 07 05 02 00 00 09 06 D5 07 .G..............
00002670 0A 02 9C FF 00 00 ......
7E 00: unknown data 0x007e (126)
05 00: Data prefix marker?I do seem to see some value before the first “prefix”, but as Mike is much further along than me, I’m going to proceed on the premise that those leading bytes are not part of the data stream. They could be padding, but let’s see if they make sense. The first thing that comes to mind is that it’s a length prefix value. But it’s far too small for what we see, we have 2018 bytes (2016 if we don’t count the “length”) a far cry from the 126 we see. let’s try dividing the ton to see of we get a sensible number.
2016 / 126 = 16
Okay well that’s a bit too specific. Seems the data block is length prefixed, but the length value is the number of 16 byte paragraphs. Testing this hypothesis on some of the other files seems to confirm this. So that gives us the following structure for the final block of data in the file.
typedef struct {
uint16_t paras;
uint8_t data[]; // paras * 16 bytes in length
} pan_cmd_data_t;
At this stage we can reliably parse and read in the file at the top level. We still have a few unknowns, but they don’t stop us from moving forward. Perhaps as we figure the next bits out, they will shed some light on the meaning of the remaining unknowns. Next up is figuring out the last large unknown chunk of data what is most certainly the animation timing and control, and this is where things get interesting. But that, my friends, will be the subject of another post.
Ouch my eye! 
Leave a reply to RedMike Cancel reply