MAYA FILE TRANSLATOR

Florian LOITSCH

Trainer: Diane LINGRAND

Co-Trainer: Benjamin PRUCHA

Summary:

During this project a file-translator for Maya (otherwise known as exporter) has been developed. This file-translator exports a given Maya-scene into an enhanced proprietary format. Wootsoft's current format was used as basis, but improvements were made.

Furthermore a small viewer has been implemented to test the correct export and import of the models.

Context

Wootsoft is a small enterprise, situated in Austria. Two years ago it started the development of “Iritor Online” (a Massive Multiplayer Online Role Playing Game). From the beginning on, Wootsoft employed several artists, who created a great amount of models in Maya (a major 3D content creation tool. See Illustration 1). Already at that time the export-process wasn't optimal: An artist (or in rare cases a programmer) exported the model into an X-file (the DirectX-specific file-format). Then a small utility was run over these files, and extracted the Iritor Online relevant information. During this process the X-file was split into up to three separate files, to share information between different models.

Shortly before the beginning of this project an internal decision had been made to port the game onto the Linux platform, and to rewrite the 3D-engine under OpenGL. While it was (and is) still possible to export models using the old process, it is clearly not the ideal way to go: X-files are Microsoft specific, and are not supported under Linux. It is hence impossible to convert an X-file to an IOM-file (Wootsoft's proprietary file format) under Linux. Furthermore, the switch from DirectX to OpenGL induced a switch from left-handed to right-handed coordinate-system. Therefore an adapted converter is needed. Wootsoft finally decided to assign this project to me to write a native Maya-exporter, that would export into a new IOM-format.

Illustration 1 a Screenshot of Maya

Requirements

The project was split into two major parts:

• the Maya-exporter (in the Maya-language a “file-translator”) and C++ importer.

• a platform-independent fast small model-viewer, that allows to inspect the exported data.

Maya-exporter

Maya (by Alias Wavefront) is a well-known 3D content creation tool, that exists on several platforms: MS Windows, Linux and MacOSX. By providing an open C++ API and a powerful scripting language (MEL), Alias Wavefront invites programmers to write plug-ins for Maya.

The C++ API interfaces directly with the Maya-internals, and MEL gives a way to write platform-independent UI-dialogs. Using Maya's shared libraries, it is also possible to write stand-alone executables that work on Maya-scenes. This allows small tools without the need of installing the full Maya suite.

As Wootsoft wasn't satisfied with their current proprietary file-format a new one had to be developed. While the initial idea of sharing data between different models seemed to be good, it was actually never used and made things more complicated. Consequently, the new file-format should contain all necessary information for a model in one file. If possible the file-format should provide the possibility to add eventually textures to it, so a model-file would be self-contained.

Once the file-format stabilized, an Maya-Plug-in had to be written. This plug-in should interface with a small MEL-script, that allows the user to change several options. Those options would depend on the final file-format, and the exporter, but it should be easy to add/remove new choices for Wootsoft-employees.

Maya provides a way of writing stand-alone programs (using their API), and it would be nice to have a small tool, that extracts the model out of a Maya-file without starting Maya. While this isn't necessary, it would be nice for batch-jobs, etc.

During the project another requirement emerged: it has been decided that the exporter was going to be Open-Source. As a consequence the code should be easy to read, and easy to expand. While this should be a goal of every software-project, freeing the source adds of course additional pressure.

Illustration 2 a static mesh

Illustration 3 bound to its Skeleton

Illustration 4 animating the skeleton influences the mesh

In the next paragraph I'm finally presenting the requirements for the exporter itself and explain the exported data. Iritor Online works with two different model-types, which need to be exported differently:

• Static models (see Illustration 2)

  Static models are clearly the easiest to export: the exported data consists of one big mesh, where each vertex of that mesh contains the following elements:

• A position in world-space.

• A normal. If a vertex has two different normals, the vertex is exported twice.

• UV-coordinates (aka ST-coordinates under OpenGL). Again, if a vertex contains two different sets of UV-coordinates, the vertex is exported twice.

• Animated models (see Illustration 3 and 4)
  Animated models are way more complex to export: three components are extracted out of the Maya for them:

• the skeleton: Instead of animating all vertices, a simpler bone-structure is created, onto which the vertexes are attached. Artists only animate those bones, instead of the whole set of vertexes.

• a skin: this is an enhanced version of the previous mesh (for the static model). As every vertex is affected by up to four bones, additionally to the already exported elements, four bone-indices and four bone-influences are exported. The bone-index points to the influencing bone, and the influence determines the force this bone is affecting the vertex.

• the animation-data (using key-frames): The animation-data consists of simple functions. For key-frames (that the artists carefully choose during their model-creation), a transformation-matrix is exported. Using these matrices the bones can be animated. As the skin is attached to the bones, a complete animation is exported this way.

In addition to the listed items, an additional “shadow-mesh” needs to be exported for most objects. Contrary to the previously cited meshes a shadow-mesh doesn't contain UV-coordinates. Furthermore a shadow-mesh needs to be “closed”. This means, that, even after moving a vertex no gaps are allowed. While the exporter can't modify the mesh to satisfy these conditions, it can enforce them, and warn the artists in case of problems. Often helpful advices (like the position of the problem-vertex) are appreciated.

A Maya-scene stores other useful information that could be used at the game-side. For instance the texture-filenames are contained in every Maya-file. Optionally the exporter could include as much as possible of these information. As it wasn't clear, what data could be extracted out of Maya, the file-format had to be kept very flexible.

Model-Viewer

Before this project started Wootsoft used the game as model-viewer. Every time a model needed to be inspected the model was included, and then viewed in the game. In the beginning this hadn't been a problem, as loading time had been short, and it hadn't been very difficult to add new objects to the game.

Over time the load-time increased, and viewing models became cumbersome. As the C++ import functions needed to be tested anyways, a small viewer should be created.

Planning

Illustration 5 and the following table illustrate the planning, that has been surprisingly well worked out. No major derivations have been encountered.

Completed Work

The completed work can be split into 3 major parts: the time spent on finding a new file-format. The work done on the exporter itself, and finally the Viewer.

.IO42

The new file-format (using the extension IO42, for Iritor Online and the magic number 42), isn't actually very complicated, and still very open for changes. But this was a goal during conception.

Instead of defining a whole new file-format, that nobody would ever be able to read without a proper importer, I decided to use a zip-library to combine several smaller files. So instead of having (up to) three different files for one model, as has been the case with the old file-format, there will only be one.

This gives several advantages over the old format:

• One self-contained file for every model.

• Expandable. There is really no difficulty in adding new data to a model. The data could be added during the export-process (like texture-names), manually (like TODO-files, comments) or automatically during a batch-process using other ZIP-tools.

• The contained files are simple and hence easy to read. For instance, a texture can still be viewed without problems, using a simple Image-viewer.

• Several models could be grouped into one bigger model. This feature hasn't been used yet, but one could easily join several models, by either creating sub-folders, or just including (recursively) the model-files into a new zip-file.

• The data is compressed, and hence uses less space, which is especially nice for the non-compressed TarGA-files, that are usually used as textures.

A simple example of such a file is shown in Illustration 6, where jorgalosk.io42 contains a mesh-file, animation-data, and a texture.

Too make the transition from the old format to the new one as smooth as possible it has been decided to use Wootsoft's old formats within the Zip. On the one hand this allows the creation of new object-files without even exporting (just drop the existing files into a new Zip). On the other hand it's not possible to use standard 3D viewers to inspect the meshes. If for instance Maya's .obj format had been used many free tools could have displayed the model.

To implement this new file-format the exporter needs to be able to save information into a ZIP-file. Fortunately I found (after some research), a BSD-style ZIP-library, that suits perfectly our needs: zlib. As this library is heavily used in major GNU-programs it is believed to be stable, and seems to be a good choice.

Exporter

Although Maya is a great and well-know 3D program, it was until recently very expensive, and only reserved to big companies. Furthermore these companies usually avoided publishing their exporters (and knowledge in generally) of Maya. This makes sample-source code very rare, and difficult to find. While the Maya-API is well documented, the documentation rarely exceeds a JavaDoc-style: every class is documented, but big concepts, and “standard ways of doing” aren't explained anywhere. The provided code-samples are next to useless for my exporter, and due to the richness of the Maya API there is always more than one way of doing something (although not always all ways are correct).

Most of the time has hence been spent on researching more information, and on trying different ways of exporting.

Static meshes are needed by Maya itself, and the interface to extract them is relatively well documented. After I had found ways of extracting static meshes including their normals, and UV-sets, animation-data was the next step. This showed itself more difficult than foreseen. As seen in Illustration 8 which shows Maya's internal representation of a simple scene (Illustration 7), Maya uses nodes to manage its information.

Furthermore developers can add their own nodes into the scene. While this allows to write great plugins, it also makes exporting very difficult. Even Maya itself doesn't know, what each node does, and how it influences the scene. When writing an exporter, one can prepare for all known nodes, but a new plugin that introduces previously unknown nodes could break the exporter.

There is of course data, that Maya needs, and where Maya provides interfaces to get this information (for instance mesh-information), but animation is not completely part of it. Maya can extract a mesh at every specific instance, but it can't always tell, how a mesh has been influenced. As Wootsoft needed keyframes, and skeleton-information, the standard Maya-interfaces weren't sufficient. It is still possible to extract those information, but any new plugin could change the scene in ways, that break the exporter.

Unfortunately all found exporters had the same problem. And they all dealt differently with it. MS DirectX x-file exporter, for instance, just added code over code. Their exporter contains about 7500 lines, where one of the more important functions itself measures 2000 lines (which made reverse-engineering very difficult).

Exporting animation requires hence a thorough understanding of Maya's internals. The exporter I delivered is not complete (which is close to impossible), but succeeds in at least as many cases, as all other open-source exporters.

As the exporter should become Open-Source, I tried to make the code as clean and extensible as possible. The main-function is only 4 lines long, but shows the clear separation of all tasks. This might seem obvious, but usually exporters write the data instantly after they have extracted (and optimized it). My approach certainly needs more RAM, but knowing, that Maya itself uses about 500MB for itself this shouldn't be a big issue.

void Export()
{
Scene extractedScene = extractScene();
optimizeScene(&extractedScene);
convertScene(&extractedScene);
writeScene(extractedScene);
}


The extraction-phase does just that: it extracts all information the user wants to export (and the exporter is capable of).

During optimization faces are merged and vertices collapsed. This could have been done during extraction, but I wanted to keep the tasks separated as much as possible. Other projects might want to add further optimizations, and here's the right place.

Different projects might use different coordinate-systems. Wootsoft itself switched from DirectX's left-handed coordinate-system to OpenGL's right-handed coordinate-system. In order to help other projects to convert directly into the right system, the convertScene function converts into the right system (only a short string like “+x-y+z” is needed). Conversion of units is done at this time too.

Finally the whole scene is written into the file. In our exporter this uses the libz-library to create a Zip-file, but if somebody wants to store it differently only the write-method needs to be adjusted.

Options Dialog

Although only a smaller part of the whole project, the absence of the options dialog was quite annoying. There was no reasonable way of telling the exporter what to export, and what not. Using Maya's scripting language MEL, a small dialog exposing the given options has been created.

Actually this task was not very difficult. MEL is written for artists, and hence quite easy. The only real difficulty lied in the fact, that Maya had to be restarted, when a bad script was loaded.

See Illustration 9 for an example of the dialog.

Model Viewer

Illustration 10 Screenshot Viewer

Illustration 11 Screenshot Viewer (Jorgalosk)

After I had successfully exported my first model, Benjamin Prucha sent me their graphics-engine, so I could start implementing the viewer. Although I first encountered some starting-difficulties (the new code wasn't yet very stable and easy to compile), it soon enhanced into a working state. I then added the code to load static models, which worked seamlessly.

At that moment the Viewer was integrated into Wootsoft's production environment, and I subsequently only needed to provide the changes I made on the C++ importer (animation,...). Wootsoft's employees (and in particular Peter Reiterer) then improved the viewer on their own.

See Illustration 10 and 11 for some screenshots.

Future Work

A project that is used in a production environment is nearly never complete. Feature-requests are made, incompatible scenes are discovered and plain simply bug-reports are submitted.

But beside of this never ending future work, there are some points, that are already on the TODO-list:

• I used C++ exceptions to signal errors within the exporter. Unfortunately they aren't working correctly under Linux. In fact the exception is thrown, but the catches are just letting the exceptions through, and subsequently crash Maya.
  This is a major bug, but until now, I haven't found any solution.

• Currently the exporter only extracts (and hence exports) animation-data, if there's only one animation in the scene. Wootsoft splits their animations into multiple Maya-files, and this isn't yet a problem. But the exporter is supposed to work for other projects too, that store multiple animations in one scene.

• Although the exporter correctly extracts the texture-name, the texture is not yet copied into the Zip. This isn't a very difficult task, but there has always been something more important on the list.

• Only one UV-set is exported. Again, this is not a problem for Wootsoft, as they are only using one set, but other projects could benefit from this feature.

Personal Conclusion

Knowledge is Power. Computer Science is one of those domains, were this proverb applies perfectly. The code for the current exporter (admitted still not feature-complete), is not longer than several hundreds of lines. It's amazing how much time has been spent on finding those lines. Wootsoft (and especially Benjamin Prucha) helped me quite often, but sometimes I was just stuck for several hours, trying to find a simple line to be able to continue.

On the other hand, developing the new file-format, and discussing it with Wootsoft's employees was quite fun. Creativity was much more important during this process.

Even the local distance didn't harm during this process and the project in general: Thanks to the Internet, and all its ways of communication Wootsoft's employees were always available and every question was quickly answered.

Finally, I've to confirm once again, that most projects just seem smaller than they are: Although I distributed the tasks generously during the creation of the planning, I often had difficulties staying on time.

References

David A. D. Gould “Complete Maya Programming”, Morgan Kaufmann, 2003

http://www.alias.com/eng/products-services/maya/index.shtml (Alias Wavefront's Maya)

http://www.ewertb.com/maya/api/ (Maya API How-To page)

http://www.greggman.com/pages/mayastuff.htm (Maya API Tips, Hints and other random stuff)

http://www.gamedev.net/reference/programming/features/mayaexporter/ (a sample exporter)