Shape from Shading

Shape-from-shading (SFS) has been an active subject of research for over two decades, and may be regarded as one of the classical problems of computer vision. In recent research we have developed a SFS technique based upon the variational approach of Horn and Brooks [20,21,22]. Our scheme addresses one of the main problems with the Horn and Brooks technique - its tendency to over-smooth the recovered needle-map, leading to a loss of detail in regions where the surface orientation varies rapidly. Several other solutions have been proposed to this (e.g. [23]), but our research has shown that the apparatus of robust statistics may be applied to the problem with encouraging results [17,18].

In brief, we wish to solve the normalized image irradiance equation

E(x,y) = R(p,q) (1)

where E(x,y) is the image of the object, and R(p,q) is the reflectance of a surface patch oriented such that its normal has direction n=(-p,-q,1)^T. The quantities p and q are the components of the surface gradient in the x and y direction respectively, i.e. $p = \frac{\partial z}{\partial x}$ and $q = \frac{\partial z}{\partial y}$.

If the surface is assumed to have Lambertian reflectance properties, the brightness of a patch will simply be proportional to the angle between the surface normal and the light source direction, s. The image irradiance equation then becomes $E(x,y) = {\mathbf n}\cdot{\mathbf s}$. Unfortunately, this is under-constrained for the recovery of p and q over most of an object's surface. Hence, we must introduce an additional constraint on the smoothness of the recovered needle-map. This is encoded by constructing an energy functional of the form

$$I=\int\int \Big(E(x,y)-{\mathbf n}\cdot{\mathbf s}\Big)^2\non... ...l \mathbf n}\over{\partial y}}\right\Vert \right)\right) dxdy$$

where $\rho_{\sigma}$ may be any regularization function, and $\lambda$ is a Lagrange multiplier. The first term of this functional encodes the image irradiance equation. The second term uses the derivatives of the recovered normals to penalize sharp changes of orientation according to the function $\rho_{\sigma}$.

Applying the calculus of variations and discretizing the resulting Euler equation, we develop the following generalized update equation for iteratively estimating the surface normals

$${\mathbf n}_{i,j}^{\left(k+1\right)} \Big(E-{\mathbf n}_{i,j}^{(k)}\cdot{\mathbf s}\Big){\mathbf s}$$ =

$$\frac{\lambda}{2}\left\Vert{{\partial {\mathbf n}_{i,j}^{(k)}}\ov... ...rt{{\partial {\mathbf n}_{i,j}^{(k)}}\over{\partial x}}\right\Vert\right)\times$$ +

$$\left({\mathbf n}_{i+1,j}^{(k)} + {\mathbf n}_{i-1,j}^{(k)} - \le... ...l x^2}\right){{\partial {\mathbf n}_{i,j}^{(k)}}\over{\partial x}}\right)\Bigg]$$

$$\frac{\lambda}{2}\left\Vert{{\partial {\mathbf n}_{i,j}^{(k)}}\ov... ...rt{{\partial {\mathbf n}_{i,j}^{(k)}}\over{\partial y}}\right\Vert\right)\times$$ +

$$\left({\mathbf n}_{i,j+1}^{(k)} + {\mathbf n}_{i,j-1}^{(k)} - \le... ...l y^2}\right){{\partial {\mathbf n}_{i,j}^{(k)}}\over{\partial y}}\right)\Bigg]$$

In the quadratic case where $\rho_{\sigma}\left(\eta\right)=\eta^2$, this becomes the update equation used by Horn and Brooks [20]. However, any other function may be used as the regularization term, and we have investigated several robust measures, including the classical Tukey [24] and Huber [25], and the Adaptive Prior Potential Functions of Li [26]. We also introduced [18] a continuous version of the piecewise Huber robust estimator, described by

$$\rho_{\sigma}\left(\eta\right) = \frac{\sigma}{\pi}\log\cosh\left(\frac{\pi\eta}{\sigma}\right)$$ (2)

and found that this yielded the best results by offering a compromise between over-smoothing and noise rejection/numerical stability.