Interesting problem! Assuming you have pairs of corresponding points (e.g. you know that points x1, x2, x3, etc. before the rotation map to y1, y2, y3, etc. after the rotation), you can find the center of rotation using geometry. For each pair of corresponding points x and y, find their midpoint; then draw an infinite line perpendicular to the line between x and y, through the midpoint. The center of rotation must lie somewhere on this line. If you have multiple pairs of points, the lines constructed from all of them should converge (at least roughly) at a single point. Intuitively, the more pairs of points you use, the better an estimate you’d be able to get for the center.

Now you just need to figure out how to get the “average” convergence point of a set of lines. You could maybe look at all the intersection points of all pairs of lines and find the centroid of those, but that sounds expensive since it’s O(n\^2) in the number of lines. There’s probably a better way, but I can’t think of it at the moment.

Once you have the center, you can then find the angle of rotation by again considering each pair of points and calculating the angle between them relative to the center. Then averaging all the angles should give the best estimate for the true angle of rotation.

I am writing a routine that removes camera motion (such as camera shake) from digitized video. I am using least-mean-squared to track points, which works well enough. I am able to track a point in the video and use it to remove the camera motion.

I would also like to remove rotation (bank) of the camera, as the camera may be handheld and have a small amount of banking. Unfortunately, I don’t know the center of the rotation. It may be near the center of the camera (center of the screen), or it may not be.

I can track several points, say a triangle, and I was hoping this would help me determine the center of rotation. However, I have no idea how to go about actually finding the center of rotation using these points at the moment.