The ordinary (arithmetic) average of the works fine for multiple vectors. What people usually do is add up the unnormalized normals for each of the triangles sharing a vertex, and normalize the result. This gives the effect that triangles with a larger area contribute more to the vertex normal, which is a nice heuristic.

But all this is just a heuristic. If you want to be mathematically rigorous about it, a triangle mesh isn’t differentiable along edges or at vertices so one can’t define a meaningful tangent or normal there. However, if the triangle mesh is an approximation to some ideal curved surface that is differentiable (for instance a Bezier patch or NURBS surface), then the vertex normals ought to be calculated based on the tangent vectors to that surface (its parametric derivatives). On the other hand, sometimes you deal with something like a subdivision surface where the definition is too complicated to differentiate explicitly, or you just have the triangles and you don’t have a description of the original Bezier or NURBS patches. Then falling back to this kind of heuristic is about the best one can do.

Feel free to come up with your own ‘average direction’ heuristics, if you’re uncomfortable with the standard ones - but realistically, the arithmetic average works just fine for most situations.

Ok, I’m “stuck” on a very basic problem: A couple of triangle’s share a vertex, what normal vector should I set for it. That is; what do I call the “average direction vector”?.

Everyone seems to just normalize the vectors, and then simply find the average of the resulting cartesian components. Alternatively, the result of this could be normalized.

For only two vectors, the average direction vector should of course lie in their plane, and it should bisect the angle between them.

Normalize the two vectors to get an isosceles triangle. Their directions, and hence the average direction vector, does not change as a result of this. Now the vector vb - va is the base of the triangle. And because the triangle is isosceles:

va + (vb - va) / 2

lies on the line bisecting the angle between them.

This can be rewritten as:

va + (vb - va) / 2 = vb / 2 + 2*va / 2 - va / 2 = vb / 2 + (2*va - va) / 2 = (vb + va) / 2

Wich is just the average of the cartesian components of the vectors, so for two vectors, what everyone seems to use works.

But what is the definition of the “average direction” of more than two vectors?

My best bet thus far is the vector wich minimizes the sum of the “errors”. Where an error is the angle measured from the average to the different vectors, but I’m not sure…