Optical flow techniques

Optical flow techniques are based on the idea that for most points in the image, neighbouring points have approximately the same brightness. In other words, the world is made up of continuous objects over which brightness varies smoothly.

Let I(x,y,t) be the irradiance at time t at the image point (x,y). If a little patch of uniform brightness around (x,y) does not change with time, then $\frac{\partial I}{\partial t} = 0$, implying

I_x dx/dt + I_y dy/dt + I_t = 0.

Now let u = dx/dt and v = dy/dt. Using the values of u and v at a grid point (i,j) and its neighbours, we can measure a departure from smoothness by

$$S_{ij} = \frac{1}{4}\{(u_{i+1,j} - u_{i,j})^2 + (u_{i,j+1} -... ...,j})^2 + (v_{i+1,j} - v_{i,j})^2 + (v_{i,j+1} - v_{i,j})^2 \},$$

while any error in the optical flow constraint equation is given by

C_ij = (I_xu_i,j + I_yv_i,j + I_t)².

So we want a set of values u_i,j and v_i,j that minimize

$$E = \sum_i \sum_j S_{ij} + \lambda C_{ij},$$

where $\lambda$ is a regularisation constant.

We differentiate E with respect to u_kl and v_kl to get

$$\frac{\partial E}{\partial u_{kl}} = 2(u_{kl} - \overline{u_{kl}}) + 2 \lambda (I_xu_{kl} + I_y v_{kl} + I_t)I_x,$$

and

$$\frac{\partial E}{\partial v_{kl}} = 2(v_{kl} - \overline{v_{kl}}) + 2 \lambda (I_xu_{kl} + I_y v_{kl} + I_t)I_y,$$

where $\overline{u}$ and $\overline{v}$ are local averages of u and v.

We equate these expressions to zero to get

$$(1 + \lambda I_x^2)u_{kl} + \lambda I_xI_yv_{kl} = \overline{u_{kl}} - \lambda I_xI_t$$

and

$$(1 + \lambda I_y^2)v_{kl} + \lambda I_xI_yu_{kl} = \overline{v_{kl}} - \lambda I_yI_t.$$

This gives us two equations in the two unknowns u_kl and v_kl. These can be solved directly and suggest the iterative scheme

$$u_{kl}^{n+1} = \overline{u}_{kl}^n - \frac{I_x \overline{u}_... ...I_y \overline{v}_{kl}^n + I_t}{1 + \lambda (I_x^2 + I_y^2)}I_x$$

and

$$v_{kl}^{n+1} = \overline{v}_{kl}^n - \frac{I_x \overline{u}_... ..._y \overline{v}_{kl}^n + I_t}{1 + \lambda (I_x^2 + I_y^2)}I_y.$$

In other words, the new value of (u,v) at a point is equal to the average of the surrounding values minus an adjustment in the direction of the brightness gradient.

Note that this scheme also requires estimates of the values I_x, I_y and I_t. These can be computed by taking local averages in neighbourhoods about the grid point (i,j,k).