Suppose we have an estimate a of the minimum of . We can refine this by choosing some direction vector u and searching for the minimum along the line . This is a 1-D minimization to find the value of which minimizes the function . a is then replaced with .
The simplest algorithm for locating the minima of is to repeat this process, cycling through the n unit direction vectors which span the parameter space. However, this can prove to be very inefficient if we end up making many small steps, zig-zagging along a valley, when it would be better to move in directions aligned with the valley (Figure 3).
Figure 3: Minimizing along each unit direction in turn can be inefficient.