Mafia II Import/Export filter guide

2k Czech Mafia II game modding.

Moderator: Oleg

Mafia II Import/Export filter guide

Postby Oleg » Wed Jul 27, 2016 1:39 am

Mafia II models import/export guide.

Preamble.
This game exploration took a really long time starting a late 2010 with a raw and buggy importer created for ZModeler2 (that was never released to the public). The second attempt to create an importer for ZModeler3 was made in early 2015 (succeeded, but was never released) and was interrupt by research of GTA5 formats. And finally winter-spring 2016 an exporting code was developed with constant tryouts to fill the gaps in format to make modding life easier. Unfortunately, this game formats are still far from fully decoded, so the modding abilities are very limited with a lot of constrains one have to respect to get the modded content run in game.

Original guide was written in Russian language and can be found here:
http://forums.playground.ru/mafia_2/tec ... _2-903602/
You can try Google translation, but it's not that good.


Tools.

SDSTool.exe

Game files are packed in .sds archives, so the first tool you need is an archive packer/unpacker. I personally use a console utility "SDSTool" that requires to write packing/unpacking commands to specify operation, source/destination file and source/destination folder. There is a GUI for this tool that you might find much more user-friendly. On the other hand, using console utility to pack the same file all the time can be put into a batch file (*.bat), including a line of code to copy packed file into the game folder, meaning you run a single file to pack and copy (and run the game too).

SDS tools are attached to this post.

The command line for unpacking file would look like this:
SDSTool.exe e -o Orig somefile.sds
where e is "Extract" command, -o Orig is the destination folder and somefile.sds is the file to unpack;
Once file is unpacked, you can create a copy of "Orig" folder and name it "Mod", so you get identical folder contents to pack. Modding will involve updates in just few of files in "Mod" folder.

The command line for packing file would look like this:
SDSTool.exe c -o somefile.sds Mod\Stream.xml
where c is a packing command, -o somefile.sds is the destination file (could have a full path/name to game folder if needed) and Mod\Stream.xml is the path to a folder with unpacked content and a content-definition file "Stream.xml" used to re-create .sds file. The Stream.xml is created during unpacking of .sds file and you are allowed to edit it to make new entries there (e.g. to inject new textures, sounds, or remove unnecessary files).

MafiaIIMaterialsMerger.exe
This small utility was made to inject new materials and update existing materials changes in a single materials database of the game. The game uses a single file for all materials of all models it has: MafiaII\pc\edit\materials\default.mtl
This .mtl file should contain materials for your modded content and, if you change original materials, it should have associated updates made there. The game is very sensitive to materials, as it has some materials-role assignment build in deep of code. For example, each vehicle has its own "paint/exterior" material that the game will accept for color-painting. If you create a different material, or rename this material, the vehicle will not get painted anymore in game. You are allowed to update existing material (by setting your own textures onto it), so an original material will get updated and the game will respect it when painting your vehicle. Another example, is a global "lights and stuff" material, named GPLC (Global Plate + Light + Colors) - this is the only material that the game will trigger condition-lighting on. It is used for headlights/taillights/reverselights/turn signals and for license plate numbers. You can't use your own textures on this material, as all other vehicles in game use the same material on them. Quite a big constrain I must say.

The merging utility should be placed in MafiaII\pc\edit\materials folder and will put contents of all *.mtl files in current folder into a file named default.mtl (updating content in it when matching materials are found).

Each material is identified by 64-bit hash value that the materials editor in ZModeler will keep on user-defined option named "Hash". These values should be unique (you can't have several materials using the same hash value). Original hash value upon loading a model is stored on material, you can change material if needed, and an exporter will prepare an updated material associated with the given hash value on export. Materials merger tool will update original material with the matching (by hash) on an exported model. An exporter will create a mini-base of model materials (a *.mtl file) that you put into MafiaII\pc\edit\materials folder and merge with MafiaIIMaterialsMerger utility. You can release your mods with mini-base *.mtl file, so end user can merge materials during mod installation.

Import models
In order to import a model, you should create a text file in a folder of unpacked file. Rename the text file to *.mafia, e.g. rename jefferson_provincial_z.txt to jefferson_provincial_z.mafia
The import/export filter requires a .mafia file as an entry point, that specifies the folder to deal with. The importer will examine all files in this folder to build the scene; the exporter will generate all needed files in the specified folder on export. Contents of .mafia file is insignificant.

The importer should have a path to materials library file(s) specified, as the model itself is not equipped with materials data, only reference hash values are available. You should specify location of original default.mtl file located in MafiaII\pc\edit\materials folder.

The only option you will need to care on import is "Slice to parts". This option allows to slice geometry into parts using a bones references. As a result, the model will get loaded as a set of objects on hierarchy, and some objects will get replaced with dummy nodes. In most of cases, you should load a model loaded as parts, rather than a single geometry object. It is quite possible for a model to be sliced to parts in a bad fashion, as model can contain mesh deformation bones with wrong syntax. The filter has a build-in set of mesh deformation bones for each part, but some models have bad syntax on deformation bone names. as a result, the part could get sliced to actual part and it's deformation part. good slicing produces a mesh part and dummy node instead of deformation bone.

Scene limitations
Each model is equipped with predefined set of objects, and most of them are important (can't be deleted, renamed or even moved). All collision objects should retain, while you are allowed to change their shape/size, but not position. Deformation objects (converted to dummies) must be retained with no changes in position or hierarchy placement.

You can not add new collision objects for your custom parts, as well as you can't add deforming bones (deforming dummies) for existing or new parts.

Arbitrary objects can be added to scene, as much as needed. Object names should be unique within the entire scene.

Original vehicle import can create several branches of hierarchy: one for rain-drops reflection, one for some in-game FX (probably burning fx) and a hierarchy branch with the vehicle itself. Vehicle branch should have user-defined option named "root" on the topmost dummy, or alternatively, it should go prior to all other hierarchy branches.

Materials limitations
After dozen of tests it was evaluated that the vehicle should be kept within eight materials if possible. Raising an amount of materials can lead to game crash with no obvious reason or possible fix to the moment.
Each material should have an Adaptation be assigned on it, so an in-game shader could be applied:
paint - model exterior with vehicle paint support. Includes regular texture, bump-map, damage (displacement) overlay with scratches, a dust/snow mask texture (UV#2) and additional uv-channel (UV#3) for custom vinyl livery texture. In-game color paint (primary/secondary) is assigned on per-poly level via Properties->General->External State-> Flag#1 (primary) and Flag#2 (secondary).
paintL2 - simplified model exterior with vehicle paint support for LODs. Uses regular tetxure, bump texture and dirt/snow mask on UV#2.
generic - very alike to "paint" adaptation, but has no in-game color painting, neither vinyl liveries (UV#1 and UV#2 are used only).
mesh_lit - diffuse, bump, specmap and emissive additive overlay livery on UV#2. This shader is used for vehicle lights. You should use original material "GPLC" if you want your vehicle lights to behave properly (see below).
diff - diffuse texture only.
snow - diffuse texture only and specular snow-flakes texture, alpha-blended for in-game snow fading. It is recommended to use original snow material with original hash on it, so it will work as interactive/disappearing snow in game.
basic - general-purpose diffuse, bump, specmap and envmap shader.
skin and cloth are for character support.

Materials uniqueness
Each material in game has a unique identifier (hash value) that geometry model binds to. Newly-created materials will get their new hash values based on their names, so you should give unique and distinguishing names to materials in your mods (avoid names like "Material 01" or regular names like "chrome", "black" etc). Original identifier of materials (when you import a model) is stored in a user-defined option named "Hash". If you copy material, make sure to remove this property on a copied material to avoid duplicates being share the same identifying hash value.

Some materials of vehicle have specific features in game. For example, the body paint material is allowed to get primary and secondary paint (and livery texture) and it can burn when vehicle is on fire. Another material like window/glass can crash and drop glass fragments. These features for materials can not be assigned, the game has an explicit configuration on each vehicle: hash-value A is a paint, hash value B is a glass, and so on. Literally, you should use original material's hash value on your own material to trigger behavior of original material. Perfectly, you should use original material (just assign textures of your choice). You can create additional general-purpose materials (like chrome, interior, underbody, dashboard etc.) that has no specific in-game behavior, when you have used original materials already. The lack of this limitation, is a missing in-game response of newly-created materials (the simplest example is no darkening after fire/burn).

Collision objects.
Collision object should be converted to compound for standard state "COL". It could be a dummy (box, sphere or capsule), or a polyhedron mesh. The polyhedron mesh is a convex solid volume with all facets jointed to their neighbors. There could be no open edges in such an object (being rendered in viewport in Wireframe+Outline mode, it will have dimmed semi-transparent mesh lines with no solid lines at all). You can combine several collisions (of any type) into a group when you need a complex collision volume.

Each collision volume should retain original "Hash" user-defined property on it. You can't add new collision elements and you have to retain all original collision objects.

Per-poly ID property.
You can assign per-poly ID properties (General->External State->ID) to identify specific behavior or subtype for a given polygon. These IDs are respected on just few shaders/materials. First of all, a global GPLC material uses IDs to identify emissive lights (front/rear left/right headlights, brake lights, turn signals etc.) Additionally, a license plate digit/literal placeholders have their IDs too. A "glass" material uses ID values to identify group of polygons as a single glass fragment (e.g. front-left door window, right half of windscreen, rear-right door window etc.)
glass shader IDs:
1 = [left] windscreen
2 = right windscreen (if separated from the left)
3 = front left door quarter fragment
4 = front right door quarter fragment
5 = front left door main fragment
6 = front right door main fragment
7 = rear left door main window (or rear left quarter on coupe model)
8 = rear right door main window (or rear right quarter on coupe model)
9 = rear left door quarter fragment
10 = rear right door quarter fragment
11 = rear windscreen

GPLC light IDs:
47 = front left headlight
48 = front right headlight
49 = left turn signal
50 = right turn signal
51 = brake light (central)
52 = left brake light
53 = right brake light
54 = reverse light left/right


Dummy points.
Dummies in scene could be of Box shape only and there are two types of them: solid dummy and wireframe dummy. The solid shape dummy is used as a point/locator. A wireframe shape dummy is exported as "locator and dimensions". It could specify some game-specific information. For example, a set of "climbbox" wire dummies is used by game engine to identify whether player can climb onto some volume from at given point (and if he can, what kind of animation could be played).

Geometry.
Most of in-game shaders use bump-maps and two UV channels. Default vertices format for your meshes should include tangents at 2 UV channels. Game has two lods only - the high-detailed model and a very-low detailed model (drawn at the very far distance). You can convert mesh objects to compound with states L0 and L1 on them for high- and low-detailed variants. The file format seems to have a limitation on geometry to the moment (problem unknown yet), so you should keep meshes low detailed if possible. If you save a .z3d file (with no textures stored in it) and get a file bigger than 3Mb, you will barely succeed in exporting it to MafiaII (the game will crash on model files bigger than 3Mb).

Geometry objects should use UV#2 to map onto standard dirt/snow texture. The paint game shader uses UV#3 for livery textures.

Another important limitation is a restriction to changes of object axes. It is not allowed to change object axes when you replace the vehicle, the pivot point of bonnet, doors and trunk should retain. When you convert a vehicle for replacement, you should pick the destination model for replacement in a proper way, to match these pivoting points as close as possible. You should prefer matching of bonnet and front doors (opened in game quite often) and can get a minor mismatching on rear doors and/or trunk.


Export.

In order to export scene, you should create a folder and specify an export file name with .mafia extension. The exporter will create a set of files for your scene: FrameNameTable, FrameResource, VerticeBufferPool_*, IndexBufferPool_*, ItemDesc_*, where * means "multiple files could be created". These files can be used to replace original files in SDS pack. If you used to create a "Mod" folder containing a copy of original unpacked files, you can export into Mod folder and get these files rewritten on export automatically.

Additionally, a export_file_name.mtl file will be created - the materials configuration mini-base. This file contains definition for your scene materials and it should be injected into game's materials database on end-user installation. You can include this file in your mod as-is to let end-user install materials with MafiaIIMaterialsMerger.exe utility.

When scene or materials are not set properly, you will get an error in messaging bar of ZModeler and an export will be interrupt. You should always check the messaging bar after export to find out whether export succeeded or not.
Attachments
Mafia2ModelExample.jpg
SDSTool.rar
(183.88 KiB) Downloaded 115 times
MafiaIIMaterialsMerger.zip
(10.03 KiB) Downloaded 127 times
User avatar
Oleg
Site Admin
 
Posts: 7342
Joined: Fri Feb 06, 2004 3:54 am

Return to Mafia II

Who is online

Users browsing this forum: No registered users and 2 guests