What exactly do you mean with “common hull”. Do you mean just the intersection (e.g. boolean ‘and’) of the two convex hulls?

That should be fairly easy using some good old CSG algorithms :). Just take the set of planes that define the convex hull for object A, and use them to cut away pieces from object B. You end up with the intersection of both objects.

I need to calculate the common hull which is shared by two intersecting convex hulls.

Is there any open-source libraries that would do this?

Or a paper explaining how to do this?

I have googled the net but nothing came up.