Eliminate pixels whose associated depths are not valid due to the limitation of the range finder used for measuring the depth (mainly due specularities, poor texture, etc). These points are usually marked by the range scanner with an out-of-range number. If there are no such points we can skip this stage.
Find a localised data group inside the data space in which all the pixels appear on a flat plane. Even if there is no planar surface in the image, we can always approximate a very small local area (here 15´ 15) as a planar surface. To implement this stage and find such a data group, we choose a number of random points, which all belong to the same square of size R (this square is only for the sake of local sampling). Using these points, create an over-determined linear equation system. If the number of inliers is more than half of the size of the square, then, mark this square as an acceptable data group. The size of the square (R) is not important, however it needs to be large enough to contain adequate sample points. We set the square size as 15´ 15 in our experiments. We have chosen this size because a square of size 15´ 15 can contain enough samples. In our experiments, 30 samples were used to perform the above step.
Fit the highest model in the library to all the accepted data groups and find the residual for each point. Then, repeat the above two steps and accept the data group that has the least Kth order residual (the choice of K depends on the application [1] and is set to 10% for our experiments). This algorithm is not sensitive to the value of K. However, if we assume K to be very large, small structures will be ignored.
Apply a model selection method (here SSC) to the extended region (by fitting and comparing all models in the model library to the extended region) and find the appropriate model.
Fit the chosen model to the whole data (not segmented parts); compute the residuals and estimate the scale of noise using the technique explained in the previous section. In the next step this scale will help us to reject the outliers. It is important to note that performing this step has the advantage that it can also remedy the occlusion problem if there is any. This means that if a surface of an object is divided - occluded - by another object, we can then rightly join the separated parts as one segment.
Establish a group of inliers based on the obtained scale and
reject the outliers. We reject those points whose squared residual is greater
than the threshold T2 multiple of the scale of noise (see the
inequality r2n+1>T2
δ2 in the previous section). Then, recalculate
the residuals and compute the final scale using: 
.
Apply a hole-filling (here, we use a median filter of 10 by 10 pixels) algorithm to all inliers and remove holes resulting from invalid and noisy points (points where the range finder has not been able to correctly measure the depth mainly due to their surface texture). This step is only for the sake of the appearance of the results and has no effect on the segmented surface’s parameters because the fitting has already been performed. However, some of the missed invalid, and noisy points can be grouped in this step. This step is not essential and can be skipped if desired.
Eliminate the segmented part from the data.
Repeat the steps 1 to 8 until the number of remaining data becomes less than the size of the smallest possible region in the considered application.