31 replies to this topic

[fireside]

For animated models, FBX is by far the best I've found. It's the only "just works" format. The rest end up being some type of nightmare because someone has done something slightly differently. If you're looking for an engine, find one that uses that format like Unity.

[rouncer]

If you use obj, it basicly puts it to you to convert it to your own binary format and yes, you need to code the animator yourself.
Thanks for the tip fireside, ill check out fbx myself.

[geon]

Vilem Otte, on 28 May 2012 - 10:54 PM, said:

never, never, never ever distribute your application with text formats for meshes, textures, skinned meshes + animations, etc

Why?

Because it can be changed? That's loads of fun!

Because people can steal my art? What if I don't mind?

[Stainless]

fireside, sadly FBX doesn't "just work" TM.

Like all 3d formats it has been through a load of versions and matching the version to the loader can be a pain.

I will agree with you though that FBX is by far the closest I have ever come to the holy grail of a animated mesh format that just works.

[Vilem Otte]

geon, on 31 May 2012 - 04:16 PM, said:

Why?

Because it can be changed? That's loads of fun!

Because people can steal my art? What if I don't mind?

Nope, I don't care about anyone stealing (or re-using - depends on license) my art too... Simply because loading times for text-files is too damn high (compared to binary format) - and at some point it will take ages!

Also consider distributing editor to allow visual editing of your binary files? Is it so hard to build one (with C# it's something like 1 hour or so of clicking - with pretty icons, without ... even less).

[Alienizer]

hmmm, text files loads just as fast to me. Binary is fine, but the conversion and all, slows it down to about the same speed as loading text files. Beside, text files can easily be modified with any text editor, but binary files not, and if one byte is bad, then whole file is bad, especially a byte from the header! But again, it depent for what you use it for. OBJ and MTL are perfect in text and very fast, but textures (images) are better off as binary of course.

[Stainless]

I hate XML with a passion.

Consider a boolean inside a class saved as xml

in xml you will have something like

<class store><class><boolean>true</boolean></class></class store>

Of course there are other ways of doing it, but in general this is the sort of structure. How many bytes is that? About 60? In what other computing field would you EVER accept a storage scheme that increased the size of the data by a factor of 60?

In binary you will have

dc.b 0xff;

One byte.

Then there is the code to read it, to read an xml file you have to have a parser, kilobytes of code which you may or may not have to write yourself

In the case of the binary you can read a whole structure in a single line of code.

We spend ages coming up with compression schemes for this asset or that asset, then start throwing in xml files because "they are easy to read and edit".

Why in all that's binary does anyone think that is a good idea?

[TheNut]

Both have their pros and cons. Binary does load and compresses well in comparison to ascii, but you may have to worry about endianness and 32 vs 64-bit operating systems. If you're not careful, accessing an object that points to a binary blob can cause seg faults if the data is bogus. This is a common problem I see with developers trying to maintain backwards compatibility without writing the lengthy verification checks.

I would agree that most XML serializers tend to go overboard with schema. I worked to great lengths to reduce the clutter that .NET spits out, particularly with its dumb handling of namespaces. Still, debugging and working with ascii is a treat in comparison to binary. I've combated the issue by writing my own XML library for all platforms (C++/C#/JavaScript), which also supports a binary variant. With binary, the XML defines the schema of the object and the processor serializes/deserializes the data accordingly. When I need to debug, I just flick the switch to ascii and see what's going on in realtime. When I need to make a release build, I switch to binary to get all of its benefits with a convenient data verification process built in.

[geon]

Stainless, on 04 June 2012 - 09:26 AM, said:

I hate XML with a passion.

I feel your pain. It's not the only text based data format, though. JSON is getting more and more popular as a XML replacement even outside web development. Your example would look like this:

myObject{
  type: "store",
  myVariable: true
}

Pretty readable, and lean. MUCH better than XML.

[Stainless]

@TheNut

In the past I used a build system that allowed me to work in text and generate binary objects at build time. That handled all issues of endianness etc.

Now I can't use that system, I don't know until run time which endianness I'm running on, but I handle that with having two file readers in the code.

BinaryFileReaderl and BinaryFileReaderb

I don't have to worry about 64 bit systems.

Even this is far better than XML.

@geon

I agree there are other text based file formats, but the pervasive nature of Microshaft has made XML the most common.

[Vilem Otte]

@Stainless & TheNut

Right now I don't have to worry about endianess at all:
1.) We're working just (and only) on and for x86 and x86_64 platform
2.) When starting application (not in current version - but it'll be adopted to it sooner or later) I test sample binary file for endianess (which is quite easy to do) - if I find out it's BigEndian I'm working on, I simply use different function pointers for loading

@Alienizer

Obj and Mtl are fast, until you come to point, where u need to load some 2 MTris mesh with dozens of different materials and hundred of textures ... in this case pre-compressing + loading as binary is a huge win (storing this as text files has no point - it takes ages to load). Note, not that we currently have this-large meshes in our project .

On the other hand, we stream lots of stuff ... meshes, textures, animations ... everything is loaded on the fly - of course there aren't many other options for large-scale open world, so quick loading is important for us (and imagine that some clever artist do so cool thing like having one world tile some 30 MB of raw data (e.g. how large they'll be in memory - in fact we don't have them compressed (except for BC compression on textures)) and it's neighbour some 300 MB ... still, we have to load it quickly (note that loading 300 MB can take up to few seconds, and we don't want to lag whole game -> loading just important stuff sequentially (physics, game data - some 0.1% of all space) and streaming the graphics stuff on another core - but also, we need it quicky, coz player wants to see high detail stuff a lot sooner than when he comes near it).

Nah... I would be glad if there were some good articles on this stuff.

So in the end ... it really depends on what actually is your project.

[Alienizer]

Stainless, on 04 June 2012 - 09:26 AM, said:

I hate XML with a passion.

Consider a boolean inside a class saved as xml

in xml you will have something like

<class store><class><boolean>true</boolean></class></class store>

Of course there are other ways of doing it, but in general this is the sort of structure. How many bytes is that? About 60? In what other computing field would you EVER accept a storage scheme that increased the size of the data by a factor of 60?

In binary you will have

dc.b 0xff;

One byte.

Then there is the code to read it, to read an xml file you have to have a parser, kilobytes of code which you may or may not have to write yourself

In the case of the binary you can read a whole structure in a single line of code.

We spend ages coming up with compression schemes for this asset or that asset, then start throwing in xml files because "they are easy to read and edit".

Why in all that's binary does anyone think that is a good idea?

I hate XML, that's why I love OBJ, it's text. but almost as small as binary, and easy to parse and fast to load. XML need to much parsing.