Structure-from-motion(SFM), i.e. recovering both camera motion and object shapes from multiple images, has been one of the most fundamental problems in computer vision and has attracted much attention in the vision community for a long time. Among the numerous techniques proposed over the past years, a factorization method developed by Tomasi and Kanade[12] is widely recognized to be an elegant and useful technique because of its simplicity and stability. They showed that, once point correspondences through a image sequence are established, a measurement matrix constructed from observed image points can be factored into motion and shapes in a very simple manner. The recovered motion and shapes are determined only up to an 3D affine transformation. This ambiguity, however, can be resolved with the knowledge of intrinsic parameters of the cameras.
Although their work was based on the orthographic projection model of cameras, extensions to weak perspective[16], paraperspective[7] or even perspective[1,11,13,5,15] projections have been made in recent years. The purpose of this document is to describe mathematical background underlying in factorization-based techniques under various projection models. We will focus only on the geometric aspect of the SFM problem and will not go into how to track the features through image sequences. Therefore, point correspondences between images are assumed to have already been correctly established.