Deciphering Blade Runner

March 7, 201518Technology

I have been doing some research into voxel rendering techniques as of lately. While the whole approach has fallen to the wayside a bit with the advent of 3D accelerators, I still find it to be a uniquely elegant approach to modeling and rendering objects in software. As part of my research, I thought it would be interesting to look at how some of these techniques were used in commercial games, back in the days of yore, during the 90s.

One of the games that used voxels extensively was Blade Runner, a game I have previously written about. Westwood studios, the game’s developer, was doing a lot of work with voxels back then, Command & Conquer – Tiberian Sun being another prominent example.

As a first step, I wanted to see if I could get find the voxel model data in the game’s resource files. Like many other Westwood games, Blade Runner uses an in-house format (MIX), which acts as a container for multiple resource files. Looking at the executable at run-time, I saw that the game always first attempts to load a file from disk, and then, only if it cannot find it, get it from one of the MIX files. I’m not sure yet how it decides what MIX file to load the data from; that may just be hard-coded in the executable.

As said before, the MIX format was used by most Westwood games of that period, and has over the years been documented extensively by fans and modders of the games. In essence, each MIX file starts with a header that contains entries for each file in the container. Each file entry consists of a file ID, the offset in the MIX, as well as the file size. The ID is can be computed as a hash of the filename using the following algorithm:

// File is up to 8 characters + 3 characters for the extension
function computeHash(filename)
    filename = toUppercase(filename)
    filename = pad filename with 0x0 so length is a multiple of 4
    hash = 0
    for each group of 4 bytes
      hash = rotateLeft(hash) + group
    loop
    return hash

So far so simple – however, that means that by just looking at the MIX file, there is no way to figure out what each container entry was called, or for that matter what type of file it represents. Looking through the resources contained in the game folder, I was nonetheless able to extract some of the original file names. This in turn let me to figure out some of the data types and formats.

Here is a preliminary list of file formats used by Blade Runner:

  1. VQA files – these are videos in a proprietary format, which has been documented extensively. However, when I tried to load some of the videos that come with Blade Runner, I didn’t have much luck. The game may be using a slightly different version of the file format than the documented cases.
  2. SHP files – these contain one or more sprites. The SHP format has been used and documented all the way back to the original Command & Conquer, but as with the VQA files, the format used by Blade Runner is slightly different: instead of the more complex compression used in earlier versions, in this iteration the file starts off with a counter of images. Immediately following, each image starts with a header that gives the width and height of the image, as well as the length of the data so that one can find the start of the next image header in the file. The image data itself is uncompressed rows of 16-bit High-Color values.
  3. SET files – as far as I can tell, these files contain information on the sets / locations seen in the Game. The file contains multiple series of records, the first of which seems to be the list of objects in that location.
  4. DAT files – these are in fact containers themselves, albeit more primitive than the MIX format. The files start with a header which contains the number of files, as well as the offset of each of them. I have not made much progress understanding the records themselves yet, unfortunately. It should be noted that the game installation directory contains a number of DAT files outside of the MIX files – presumably because of size restrictions of what could be contained in a MIX container.
  5. GAMEINFO.DAT – despite having a .DAT ending, this file is actually just a collection of resource names. For example, it contains the names of all sets, as well as audio files in the game.
  6. TRE files – this format is very similar to DAT files, however they miss the initial magic number that identifies a DAT. As far as I can tell, they always contain just string resources. For example, the labels used by the UI are contained in one of the files.

Additionally, there are some other formats contained in the MIX files which I haven’t yet looked into further. Among these are the following:

  1. AUD – audio files
  2. TLK – speech files
  3. FON – fonts

As I was exploring these formats, I created a command line tool which let me browse the contents of the MIX files and load data from individual entries inside the container. The source of the project is available via my Github repository. It’s lacking quite a bit of polish at this time, so that it probably will only prove useful for people willing to dig into the code and make changes as needed; but hey, maybe someone else will be able to contribute some more information on the remaining formats.

That being said, so far I did not find the voxel models that prompted me to start this whole endeavor. At this point, I suspect that the data is stored somewhere in the COREANIM.DAT container, since that is being read by the executable at start-up. Unfortunately, I don’t have any information on the actual file format yet. I’m hoping to post more as I find more time to dig into the data.

18 throughts on "Deciphering Blade Runner"

  1. do you know madmoose ScummVM Fork + the bladerunner engine he is working on (last commit Feb/2015)?

    https://github.com/madmoose/scummvm/tree/bladerunner

    COREANIM.DAT access can be found in

    scummvm-bladerunner\engines\bladerunner\slice_animations.cpp

    starting with ine 101

    bool SliceAnimations::openCoreAnim() {
    return _coreAnimPageFile.open(“COREANIM.DAT”);
    }

    bool SliceAnimations::openHDFrames() {
    return _framesPageFile.open(“HDFRAMES.DAT”);
    }

    so he seems to “understand” the COREANIM.DAT format

    Reply

        1. I did look through that project’s code, but I wasn’t able to decode the VQA files from Blade Runner using their methods – I think there’s a difference in format between C&C (well, Tiberian Sun) and Blade Runner.

          Reply

          1. Try using a little tool called VQA to AVI. It’s not perfect but I was able to extract all of the cutscenes and most backgrounds (which are movies as well) using it. It’s also perfect for extracting sound. You will be able to open the .tlk files after renaming them to .mix.

          2. Thanks for the tip – hadn’t seen that tool yet. Also, good to know that the TLK files are actually MIX.

    1. I finally found some time to look into the code of the ScummVM re-implementation. As I mentioned, when I started looking into the BR resources I was mainly interested in finding some representation of the 3D voxel models in the DAT files. As far as I understand from the code, the model data is arranged in horizontal slices. This makes it possible to render each model as a series of horizontal lines on screen, without having to do any actual ray-tracing (which would be a more traditional approach to Voxel rendering).

      It’s certainly an interesting approach, avoiding a lot of the heavy-weight math, which probably was the result of the constraints of consumer PCs back when the game was released. However, it also imposes a limit on rendering objects: all objects are always drawn from the side, just facing in different directions. Within the game, you couldn’t have a scene which was drawn at an angle for example (although the model data could certainly be converted into a format which could then be display from any angle).

      Reply

        1. Yes, exactly. It is my understanding that the format is using pre-processed voxel data with a fixed angle used for each animation in a particular scene. That avoids having to store all the data of “true” voxel model and also speeds up rendering. However, it means that you cannot render the objects from an arbitrary angle.

          Reply

          1. Ok, I’ve talked the guy from “scummvm”, madmoose and he told me that the format is basically a 2d polygon like model. Every slice has vertics in a (x, z) format (y being slice #). You can see that in the animation demo in the top view. Those vertices get rotated, so you can see them from the “side”. The connection from one vertex to another (a line or 2d polygon) is filled with the color of the left vertex to the right. So you have a kind of voxelish look, because you have many vertexes. It’s kind of a sliced high polygon model to a simpler format…

            Processing this in slices has the advantages (I guess) that you don’t need to rotate of the z-Achsis. This might be the speed up that leads to pretty high res models on a 90mhz computer…

            I’m thinking about “reconstructing” the models and interpolating the voxels from the vertices… The resolution is high enough and if you close up the “holes” that could probably be seen from top, by simply closing by “last voxel” you might get a decent voxel model.

    1. I did not actually try to load the individual animations in my code, but you can look at the ScummVM fork linked to in thebert’s comment above to see it in action (SliceAnimations::open). The header of the DAT file contains an entry for each animation, which also gives you the offset in the file (line 73 of SliceAnimations.cpp).

      That being said, I’m not sure how the actor IDs from your link correspond to the individual animations – I suspect there will be multiple animations for a single actor.

      Reply

  5. Hello again.
    Yes ,the list i show you is the actors,
    i imagine all actor has several animations.
    I will try use this files for extract all the voxel model and animations.
    (Lands of lore 3 use the same engine ,voxels included)

    Reply

  6. Hi again.Only wanted add i remembered i had lands of lore 3 which also use the same voxel engine.I put on the computer and investigate a little.
    All the voxel seem be on “sliceX.wsx” being X =2,3,4 (The first datasheet seem is on other file (probably on the .DAT is installated.)
    With “VQA to AVI” you can extract all the files.
    I had no time to try test them because i was a little bussy trying to understand the other older voxel format which was used on the same time : “caviar voxel”.They are some RE but the file is no complete and i tried to understand “its first version” (when files finished on .VOX than .CAV .they are a little diferent (the visor doesnt let open the other) but maybe understanding this first i could understand more of the caviar.

    Ah,you can also view the lol3 vids (which are only a few of them: some strange on Weestwood :P) and the audio (are mp3)

    Reply

Leave a Reply

Your email address will not be published.