16 replies to this topic

[Wernaeh]

Hello everyone :)

Currently, I am messing with an automated serialization system - in detail, I am upgrading old C-style code to a more C++-like, type safe interface.

Object reconstruction and member deserialization have all been correctly updated - with one exeception: type safe deserialization of object pointers.

The basic concept of pointer swizzling / unswizzling is not the problem, though, it rather is type safety I am concerned with.

The old approach was to ensure type safety via class name strings, and an associated string based class hierarchy. Assigning a matched-type variable to a pointer then would be done with a C-style cast operator.

What I am now looking for is something that works more like this:

/* ... After creating an instance I, store it typed(!) for later lookup ... */


InstanceType &I = *new I(parameters);

deserializer.registerInstanceWithTypeAndName<InstanceType>(I, "I");


/*  ... later on, different function: typed lookup, should return a properly typed variable (no casting) ... */


SomeOtherType *p;

deserializer.getPointer<SomeOtherType>(p, "I");

I have no idea, though, how to implement both of these functions. I know I'd basically need a compile time map of pointer<->types within my deserializer. The pointer<-> name lookup then is trivial.

RTTI and dynamic_cast<>-ing does not work, since several of the objects do not have virtual functions for performance reasons (many small objects)

Keeping a naive std::vector<T*> around in the deserializer class for each supported T is also not really an option, since the list of classes to be supported is not known within the deserializer.

Does anybody have any suggestions here ?
Perhaps some template design pattern ? - I'm not an in-depth template specialist myself, I have to admit...

I already had a look at some C++ deserialization libraries (s11n, ...) but these either are overkill for my purposes, or don't support required features (classes without default constructor, ...)

Thank you for your time,
Cheers,
- Wernaeh

[Reedbeta]

If what you need is a compile time map of pointers to the types pointed to, you can implement that easily using partial specialization:

template <typename T>
struct extract_base_type {};

template <typename T>
struct extract_base_type<T *>
{
    typedef T result;
};

// Then, you can use it like:

extract_base_type<FooType *>::result    // is a typedef for 'FooType'

[Wernaeh]

Quote

If what you need is a compile time map of pointers to the types pointed to, you can implement that easily using partial specialization:

Thank you for the input, Reed :)

However, I already have the base type in both cases.

The problem is rather that I need a lookup of (base type or pointer type) <-> actual instances (which I meant with pointers in my previous post).

I need some way to add a {pointer to an instance of some class C} to my deserializer class, without losing the class information. Then, I need a way to retrieve this pointer again, given another class D, iff C is a child of D.

The constraints are:
=> The deserializer at compile time doesn't know which Cs are required
=> I can't use RTTI, since not all classes have a vtable pointer, but are PODs.

The problem per se should be compile-time solvable, I'm just not sure whether it is possible in C++.

I hope this was a better formulation.

Cheers,
- Wernaeh

[Reedbeta]

Hmm. I still don't think I fully understand where the difficulty is...can you show me what's going on inside registerInstanceWithTypeAndName?

[Wernaeh]

Well, I guess that is the problem ;)
I don't exactly know how to write these methods so that it is proper C++ code.

If one could assume that all serializable objects were virtual and dynamic_cast could be used, and if one could assume all objects derived from some interface "Serializeable", the following would do:

// Serializeable needs a VTABLE pointer for dynamic_casting
class Serializeable { virtual ~ Serializeable(); };

class Deserializer
{
     std::map<std::string, Serializeable*> RegisteredObjects;

     template<typename T>
          void registerInstanceWithTypeAndName<T>
               (T& instance, std::string& name)
     {
           // Here, the type is not correctly stored!
           // We rather rely on the VTABLE pointer and the Serializeable
           // interface - the template isn't really used here.
           RegisteredObjects.insert(std::pair(name, &instance));
     }

     template<typename T>
          void getPointer<T>(T *&pointer, std::string& name)
     {
          // Retrieve an object previously registered.
          // If the object is not pointer-compatible to the provided
          // pointer variable, the pointer variable remains unchanged.
          // Again, this code relies on the VTABLE pointer and
          // the Serializeable interface.
          std::map<std::string, Serializeable*>::iterator it =
               RegisteredObjects.find(name);

          if (it != RegisteredObjects.end())
          {
               T* replace = dynamic_cast<T*>(it->second);

               if (replace)
               {
                      pointer = replace;
               }
          }
     }
};


/* Example objects.. there are many more of these... */
/* Deserializer doesn't know these, though... */
class MyObjectA: public Serializeable { };
class MyObjectB: public Serializeable { };


/* Later, on, a possible use were: */
MyObjectA myobjecta =
    deserializeFromFile() or = external object, or whatever.
registerInstanceWithTypeAndName<MyObjectA>(myobjecta, "MyNameA");

MyObjectB myobjectb =
    deserializeFromFile() or = external object, or whatever.
registerInstanceWithTypeAndName<MyObjectB>(myobjectb, "MyNameB");

/* Note MyNameA and MyNameB are usually automatically generated */
/* by pointer swizzling / unswizzling, and act as unique object ids */
/* within the file */

/* Fill a MyObjectA pointer from the classes unserialized from the file. */
/* This binds the pointer to a previously registered object, and */
/* needs to be typesafe. */
MyObjectA *ptr;
getPointer<MyObjectA>(ptr, "MyNameA");

/* For instance, this call may not assign the pointer: */
getPointer<MyObjectA>(ptr, "MyNameB");

So, what I am looking for is some clever methods registerInstanceWithTypeAndName and getPointer that use the provided types to avoid using dynamic_cast and the Serializeable interface, without the need for manually adding containers for each supported client class T.

Cheers,
- Wernaeh

[Reedbeta]

Okay, I think I see now. You want something like a dynamic map from string names onto both the pointer and the type of the thing pointed to, and then to later look up that pointer by name and convert it to the appropriate derived type only if it really has that derived type.

That does seem tricky to do in C++. I've been thinking about it for a bit and haven't come up with any good ideas, although maybe one of the template metaprogramming guys here could crack it.

How about this: to each class that might be serialized, add a static function returning the class's name as a string. This can be generated with a macro for minimal code intrusion. Then you can store the class name in the map alongside the pointer. With a little work you can also have a static function that returns the class's ancestry (i.e. a list of its superclasses' names all the way back to the root) so you can compare derivation.

[Wernaeh]

Quote

Okay, I think I see now. You want something like a dynamic map from string names onto both the pointer and the type of the thing pointed to, and then to later look up that pointer by name and convert it to the appropriate derived type only if it really has that derived type.

Yes, exactly :)

Quote

How about this: to each class that might be serialized, add a static function returning the class's name as a string. This can be generated with a macro for minimal code intrusion. Then you can store the class name in the map alongside the pointer. With a little work you can also have a static function that returns the class's ancestry (i.e. a list of its superclasses' names all the way back to the root) so you can compare derivation.

This is similar to the old C system we had: A preprocessor extracted all class names and built the name hierarchy for us, which we then used to validate - at runtime, though - the class name associated with the incoming pointer against the class name of the variable.

The map within the deserializer itself maps names to untyped pointers, these are, on a match of the hierarchy test, converted to the proper class with a C-style type cast.

However, this is run-time security for something that could be enforced with compile-time security, and it is more C and less C++

I think I can live with the old system as well, but somehow, I think it could be better...

Thank you for your input and your time :)

Cheers,
- Wernaeh

[Reedbeta]

Well, at some level you're always going to need to check types at runtime, aren't you? Since you're deserializing at runtime, you need to check that the piece of data you just read from a file matches the type you expect it to be. I assume each serialized object is annotated in the file with its class names or some sort of class ID, right? Similarly, the map of instance names to pointers is dynamic, so I'm not clear on what exactly you think should be resolved at compile time?

[Wernaeh]

I'm deserializing at runtime, but I needn't check for a type match between stored data and reading object :)

Perhaps this is special to the actual deserialization process:

There are no factory methods, and one can't simply deserialize new objects from the file.

Rather, one deserializes but a single "root" object. This object then in turn deserializes further objects from within the file, based on "blocks" associated with this object.

For instance:

void Level::load(Deserializer &deserializer)
{
      resetLevel();

      int numEntities = 0;
      deserializer.loadInt("numEntities", numEntities);
      Entities.resize(numEntities);

      for (int i = 0; i < numEntities; ++i)
      {
            Entities[i] = new Entity(*this);
            Entities[i]->load(deserializer.blockDeserializer("Entities", i));
      }
}

This solves most problems with creating instances from the file - since the instancing is moved from the deserializer (which, as general might miss some of the context), on to the deserializing object (which has the context to create "owned" objects).

The deserializer then silently fails, if, for instance, one tries to read any parameter from file that does not match with the parameter an object requests. For instance, when deserializing a Level object from a File that does not contain a Level as root, there simply is no variable "numEntities". Thus, numEntities remains as 0, and no entities are loaded.

Basically, this means I am stored untyped, blockwise data, and reinterpreting that data based on the deserializing object, rather than assuming the stored data either matches or not matches my object.

This has certain advantages: For instance, it is rather simple to handle entity factoring. Rather than providing several factories for various occasions (f.e. a "LevelEntity" factory which correctly initializes the entity's level reference), the factory is integrated into the serializing code.

The only obvious shortcomings are:
+ Pointer handling (perhaps name resolving can also be integrated into the deserialization method? -> would be quite some work there, though)
+ Allowing for multiple-type array variables (i.e. an array std::vector<SomeBaseClass*> can't be correctly newed from the file without the lookup you mentioned)
+ Tightly-packed binary formats: These either need a header with layout information for various blocks, or some standard layout that works without a std::string name lookup for properties.

Well, I guess I'll sleep over that a bit.
Perhaps I'll need an entirely new design here...

Cheers,
- Wernaeh

[Wernaeh]

Hmm, I think I got a workaround that is half-way acceptable:

The default deserializer does not support any object pointers intrinsically.

However, there is another deserializer class template that may be chained onto any deserializer object - similarly to the block objects already in place.

This template object provides special handling for entity pointers, but only supports entity pointers that match in a single base class - in this case, the dynamic_cast can properly be used.

If the user anywhere requires more than this setup, he can write his own deserializer frontend class, which handles pointers in some other way.

I guess this is a good workaround, since in most cases, all deserialization entities come from a certain set of classes (GUI controls, game entities, ...)

I'll try it out and report back once I've any experiences to share.

Cheers,
- Wernaeh

[JarkkoL]

For retrieving the concrete type of a base class pointer you need to use RTTI or virtual functions, unless if you want to register each instantiated object and then search for the pointer from a global registry for that base class.

For serialization of pointers to base classes I use virtual functions myself. I have a class repository per base class, which contains registered classes, so at deserialization you can create object instancies by using class names that are saved to files. For pointers to monomorphic (i.e. non-virtual) classes there is one global class repository that's used instead (it's detected at compile-time based on pointer type which repository to use). So, you define classes something like this:

class bar_base

{ PFC_BASE_CLASS(bar_base) {PFC_VAR3(x, y, z);}

public:

  int x, y, z;

};


class bar: public bar_base

{ PFC_CLASS(bar, bar_base) {PFC_VAR3(a, b, c);}

public:

  float a, b, c;

};


struct foo

{ PFC_MONO(foo) {PFC_VAR4(i, j, k, l);}

  char i, j, k, l;

};


struct test

{ PFC_MONO(test) {PFC_VAR2(b, f);}

  bar *b;

  foo *f;

};

and register, save & load those classes/objects like this:

// register classes

PFC_REG_BASE_CLASS(bar_base);

PFC_REG_CLASS(bar);

PFC_REG_CLASS(foo);

PFC_REG_CLASS(test);


{

  // save data

  bar *b=PFC_NEW(bar);

  b->x=1; b->y=2; b->z=3;

  b->a=4.0f; b->b=5.0f; b->c=6.0f;

  foo *f=PFC_NEW(foo);

  f->i=7; f->j=8; f->k=9; f->l=10;

  test t={b, f};

  save_object(t, "test.data");

}


{

  // read data

  owner_ptr<test> t=read_object<test>("test.data");

}

If you like to have a look on the implementation details, you can find the code from here (src/core/class.h and class.inl files). It requires default constructor from classes though, because in general that's what you need for generic serialization lib. The code also does a bit more than pointer serialization though (:

[Wernaeh]

Thank you for your input - now I have some more ideas on functionality for the deserialization system :)

Basically,

Quote

if you want to register each instantiated object and then search for the pointer from a global registry for that base class.

sounded like a rather good idea: You need to have a single factory entry point for each (base) type you want to create during deserialization.

My layout now looks like this:

There is a non-template class UntypedFactory, which provides generic (untyped) functionality, and a class template Factory, which does the typed functionality for some type BaseType.

The user, at some point, knows he wants to instantiate an object that is required to be at least of type BaseType (this may be the type of the later object pointer). But actually, the object from the file has a type of ClassName, where ClassName is some string constant.
Additionally, the in-file object has an identifier to be associated with it, for later pointer restoration.

Yet, this means the user also knows that there is at least a specialized Factory for type BaseType. Thus, the user can call a typed method:
BaseType *FactoryObject<BaseType>.createInstance(ClassName, Identifier)

Each Factory includes a list of UntypedFactories, which has one entry for _any_ factory that builds a child class of T.
On instance creation, the UntypedFactory for ClassName is found in this list (or it may be the Factory for BaseType itself - if nothing is found: return NULL).

Then, instance creation calls an abstract method
virtual void* create(ClassName, Identifier) = 0
provided by that UntypedFactory.

This entry point is _not_ implemented in Factory (more on this later),
but does these things:

+ Create (and later return) an object that is of the exact type of the factory - i.e. creates a ClassName

+ Register the _typed_, created object with the _typed_ methods of each parent class Factory of ClassName - registration then continues recursively on further parent.

This requires some elaboration: Have each Factory not only carry a template parameter ContainedType, but also a template parameter BaseFactoryList.

This template parameter is used to provide a list of _typed_ base class factories within the Factory, and allows the registration call to retain type information on recursion.

Essentially, this means a class hierarchy is represented not by a hierarchy of factories, but by a series of same-level, but type-safely connected factory instances.

Once
virtual void* create(ClassName, Identifier) = 0
returns from the UntypedFactory, the list of instances within the typed base factory should now contain a properly typed pointer to an instance of ClassName under the provided identifier -
this typed identifier is now retrieved by lookup, and returned to the client.

Later on, the typed identifier can also be looked up again without object creation - and in any of the BaseClass factories that is compatible to the class that actually was instantiated.

Now for the virtual void* create() method: This one is implemented in a further class template deriving from Factory. Since create() calls the constructor of the class associated with the factory, there may be moments where further information is needed: such as constructor parameters. This can easily be introduced by creating a custom factory class.

On second reading, perhaps a little bit of code makes all of this more clear:

class BaseFactory
{
    std::map<String, BaseFactory*> ChildFactories;
    virtual void* create(Identifier& identifier) = 0;
}

template<typename ContainedClass, typename BaseFactoryList> Factory :
    public BaseFactory
{
    ....
    std::map<Identifier, ContainedClass*) Instances;
    BaseFactoryList BaseFactories;
    ....
    
    void registerInstance(Identifer& identifier, ContainedClass* instance)
    {
        Instances.insert(pair(identifier, instance));
        BaseFactories.registerInstance(identifier, instance);
    }

    ContainedClass* createInstance(Identifier& identifier, String& ClassName)
    {
        ActualFactory = Find ClassName in ChildFactories
        ChildFactory.create(identifier); 
        return Instances.find(identifier);
    }

    ContainedClass* findInstance(Identifier& identifier)
    {
         return Instances.find(identifier);
    }
}

template<ContainedClass, BaseFactoryA, BaseFactoryB>
     ExampleBaseFactoryList
{
    BaseFactoryA BFA;
    BaseFactoryB BFB;

    void registerInstance(Identifer& identifier, ContainedClass* instance)
    {
        BFA.registerInstance(identifier, instance);
        BFB.registerInstance(identifier, instance);
    }
}


template<typename ContainedClass, typename BaseFactoryList>
        DefaultConstructorFactory :
  public Factory<ContainedClass, BaseFactoryList>
{
    virtual void* create(Identifier& identifier)
    {
           ContainedClass *instance = new ContainedClass();
           registerInstance(identifier, instance);
           return instance;
    }
}

This allows for completely typesafe reconstruction of entities, for multiple inheritance schemes, and for arbitrary constructors.

Additionally, all factories may be encapsulated into different FactoryContext classes for ease of use - for instance, a GUI loader could have a different list of Factories than a level loader.

The only downsides:
- One extra lookup per instanciated entity (but: RTTI has that one as well if you dynamic_cast)
- Extra memory for lookup arrays. However, for a instance hierarchy of depth N, this is at most a constant factor of N, on some very small (pointer + identifier) overhead (each instance is registered once on each base type)

All over, I'm pretty content with this scheme...

Thanks again,
Cheers,
- Wernaeh

[JarkkoL]

Wernaeh said:

sounded like a rather good idea: You need to have a single factory entry point for each (base) type you want to create during deserialization.

You would have to register also instances not only created during deserialization, but also while constructing the data structures (like in my example). That can be a bit annoying requirement for the use of the lib.

Wernaeh said:

There is a non-template class UntypedFactory, which provides generic (untyped) functionality, and a class template Factory, which does the typed functionality for some type BaseType.

In the link I gave you, if you rename class_factory_base => UntypedFactory and class_factory => Factory, you will find the code oddly familiar ;) For each registered class I have an instance of class_factory template class. However, I have also class_repository template class derived from class_repository_base class, which is a static member of each base class and contains factories for registered classes derived from that base class. At deserialization, I first search for the class_repository_base by using the base class name and then search for the class_factory_base in the repository by using the concrete class name.

Wernaeh said:

This allows for completely typesafe reconstruction of entities, for multiple inheritance schemes, and for arbitrary constructors.

"Arbitrary constructors" in the sense that all your instances will have the same constructor arguments, which is essentially a default constructor ;) Ok, I give you that there is tiny bit of extra flexibility so that you can specify default constructor specifically for serialization, but I haven't yet found any practical use for it. It's trivial to add by using e.g. template<typename T> T *create() {return new T;} and specializing the function for types which requires custom construction, which is much less hassle than creating custom factories. IIRC, boost::serialization lib has that same feature, but I don't remember how it's implemented there.

[Wernaeh]

Thank you for your answer :)

Quote

You would have to register also instances not only created during deserialization, but also while constructing the data structures (like in my example). That can be a bit annoying requirement for the use of the lib.

Hmm - I don't exactly get what you mean ? Care to detail this ? :)

Quote

In the link I gave you, if you rename class_factory_base => UntypedFactory and class_factory => Factory, you will find the code oddly familiar

As I've said, that's what gave me the idea for the implementation I described ;)

I guess, the only difference is that your object repository and class repository are both contained within my factories.

Quote

However, I have also class_repository template class derived from class_repository_base class, which is a static member of each base class and contains factories for registered classes derived from that base class.

I don't fancy the idea of static registries and object repositories for my case though - the entire program is multi-threaded, and with statics, it's hard to control thread access - in particular if any thread may run its own deserialization process.

I rather have a factory context class in mind which includes a list of factories that is accepted in a given deserializer (in the old code base, this was a C-string -> static constructor lookup array).

Quote

"Arbitrary constructors" in the sense that all your instances will have the same constructor arguments, which is essentially a default constructor

Not exactly.
A templated create() as you indicated spawns instances based on some type - unless you use some temporary static where you store parameters.

A factory, however, can spawn instances depending on the spawning context. For instance, the following code assigns a correct parent object to any GUI control it deserializes:

GuiControl::deserialize

   (Deserializer<GuiFactoryContext>& deserializer)

{

    deserializer.FactoryContext.setControlParent(*this);


    deserializer.getBlockCount(numControls, "numControls");


    for (int i = 0; i < numControls; ++i)

    {

        deserializer.GuiControlFactory.createInstance

           (Controls[i], getBlockName(i), getBlockType(i));

    }


    for (int i = 0; i < numControls; ++i)

    {

        Controls[i]->deserialize

           (deserializer.getBlockDeserializer(i));

    }

}

This is useful in particular with objects that have some references in their constructor (i.e. NULL pointer safety)

The reason I'm a bit careful here is that previously, our old deserialization library allowed for many ugly things:
For instance, one could load audio resources and rendering resources from every serialization file.
Both the audio as well as the rendering system were threaded - and internally used serialization files as well. Thus, one could load rendering resources from some audio file (acquired a lock on the renderer), and audio resources from a rendering file (acquired a lock on the audio system)
=> Ouch, deadlock.

I guess that also is a difference between our serialization systems: Mine is more limited on purpose.

Cheers,
- Wernaeh

[JarkkoL]

Wernaeh said:

Hmm - I don't exactly get what you mean ? Care to detail this ? :)

I mean when you serialize your data structure, it can't know the type without RTTI/virtual call, if you haven't registered the pointer to serializer after instantiation. So you would have to manage creation and deletion of objects through serializer. E.g. if you have a pointer to a class returned from some factory function, do you have to register that to deserializer or is it already registered? Do you enforce a policy that all objects must be created with a serializer create() function, always?

Wernaeh said:

I guess, the only difference is that your object repository and class repository are both contained within my factories.

Okay, the thing with object repositories is that not all base classes should have one (in my design), which is why I have two ways to define a base class: PFC_BASE_CLASS() and PFC_REPOSITOTY_BASE_CLASS(). If you define a base class without object repository (about half of the base classes in my code), the data is serialized to the same file with the object referring to it, while for a base class with object repository, pointers to the type have name associated with each object and are stored as "external reference" to separate files. You obviously can't store all the data in a level to a single file so you need to think of a mechanism to define external references.

Wernaeh said:

I don't fancy the idea of static registries and object repositories for my case though - the entire program is multi-threaded, and with statics, it's hard to control thread access - in particular if any thread may run its own deserialization process.

Classes are registered only at program startup so static class registry is not an issue for object creation. With object repositories when loading new objects, you would have to lock the data structure for writing for the duration of checking if object already exist in the repository and for the duration of loading. Anyway, you will have to ensure that the object isn't loaded simultaneously from two threads anyway, so whatever approach you take, you would have to lock somewhere.

I don't really expect deserialization from multiple threads, thus I haven't taken any measures for it. Deserialization from multiple threads just opens a new can of worms, so for example if you deserialize a texture you have to create texture object using the renderer, but it can be that you have to ensure that the texture is created within the same thread as the renderer was created, or because you have single storage where you read the data, threads would end up fighting for the resource causing seeking which you try to avoid anyway, etc. So, it might be better to have only single thread doing deserialization, thus you can simplify the process from multi-producer/multi-consumer to single-producer/multi-consumer.

[Wernaeh]

Quote

E.g. if you have a pointer to a class returned from some factory function, do you have to register that to deserializer or is it already registered? Do you enforce a policy that all objects must be created with a serializer create() function, always?

Objects in my system can either be created with a call to serializer.create<BaseType>("Identifier", "ImplementingClassName") or can be registered with serializer.register<ImplementingClassName>("Identifier") - either of these two is necessary so pointer restoration works correctly.
Both calls register an object with all factories that could possibly have created this object.

Quote

You obviously can't store all the data in a level to a single file so you need to think of a mechanism to define external references.

External references right now are sparse and usually of specific type (resources, meshes, actual level geometry) - thus these have their own systems for deserialization in place (they don't anyways go well with a generic deserializer), and are referenced by a simple string. This string is resolved to actual pointers manually within each object's deserialization code.

Quote

So, it might be better to have only single thread doing deserialization, thus you can simplify the process from multi-producer/multi-consumer to single-producer/multi-consumer.

Even in this case, you still have to be careful that your deserializer thread doesn't additively lock any other subsystems (it needs to be a leaf in the thread usage graph) - but I guess multi-producer / multi-consumer still means some bit more worries about thread safety...


On a completely unrelated note - I've spent some time thinking about multiple inheritance and came to the conclusion that I will take it out of my initial design of the deserializer.

On the one hand, it currently isn't used anyways. On the other hand, multiple inheritance has some nasty side-effects:
For instance, if a class A derives from a class B more than once, and it's not virtual inheritance, it means this object needs to be registered with the factory for B twice with different addresses - now which of these should be returned for a given object identifier when a B typed pointer is requested ?

I think this causes more problems than it is worth...

Cheers,
- Wernaeh

[JarkkoL]

Wernaeh said:

Objects in my system can either be created with a call to serializer.create<BaseType>("Identifier", "ImplementingClassName") or can be registered with serializer.register<ImplementingClassName>("Identifier")...

Yeah, that's what I meant. It can also be a bit error prone though, but I guess you can assert during serialization if you are serializing a pointer which isn't registered. I get a compile-time error if you try to serialize pointer which can't be serialized, so it's a bit more robust in that sense.

Wernaeh said:

External references right now are sparse and usually of specific type (resources, meshes, actual level geometry) - thus these have their own systems for deserialization in place (they don't anyways go well with a generic deserializer)

What kind of issues you have with generic serializer for those types? I think it works pretty well, and e.g. for D3D9 textures I create IDirect3DTexture9 at deserialization and stream the data straight to the object, thus there's no intermediate buffers I have to use, etc. I think it's nice to have one uniform system for serialization, but would like to hear if there's something I have overlooked.

Wernaeh said:

Even in this case, you still have to be careful that your deserializer thread doesn't additively lock any other subsystems (it needs to be a leaf in the thread usage graph)

I don't see the problem. Only write lock I would have to do is when I insert the object to repository, which is essentially hash_map::insert(), and even in that case I think I can do it without a lock in single-producer case. Furthermore, if the object repository is properly used (i.e. there are no find_object() calls to it), other threads are not accessing the repository even for reads, since it's used almost exclusively during deserialization. There is no need to lock any subsystems either, except possibly for very short period of time if for example IDirect3DDevice9::CreateTexture() isn't thread-safe.

Wernaeh said:

On a completely unrelated note - I've spent some time thinking about multiple inheritance and came to the conclusion that I will take it out of my initial design of the deserializer.

I don't support MI at the moment either, because I think it's a sign of lack of understanding of object composition and misery of IObject type designs, TBH ;) I try to have relatively simple inheritance hierarchies and have seen the terrors (and been a culprit) of the terrors of virtual inheritance and diamonds of death in engine design (:

Anyway, if you absolutely want to, you can dynamic_cast a pointer to void* which of course requires RTTI (or use a virtual call) to find out pointer to the most derived type, so it's doable in those other designs but I don't really think it's necessary.