A collection of algorithms for this segmentation task has been implemented and evaluated by MacAulay [6] and a remarkable 98.3% success rate was reported on a data set of 4700 images. However, those images were obtained using a special sample preparation technique which included the use of Feulgen-Thionin and Orange(II) stains. These stains enhance the contrast of the cytoplasm to background and of the nucleus to cytoplasm, greatly simplifying the segmentation process.
Unfortunately, the stain used in the Papanicolaou slide preparation process, which has been the accepted method for cervical cancer screening for many years now, does not have such attractive features. As the cytoplasm is also stained, the nucleus-cytoplasm contrast is greatly affected. Weak image gradients along the nuclear border and artefacts in the cytoplasm add to the difficulty of segmentation.
More recently, McKenna [4] reported an 89% rate of `acceptable' segmentations on a dataset of 821 Pap stained nuclei images, where acceptable was defined as a maximum perceived delineation error of 5% of the nuclear area.
These previous studies have mostly relied upon the traditional methods of cell segmentation, thresholding, and edge detection with post-processing, although some authors have concocted interesting algorithms for the task [5][3].
Mathematical morphology has been applied to the problem, but results have not been encouraging. Meyer suggested its use in [7] to classify a sample of cell nuclei, however the accuracy of border delineation was not examined. Lee et al [8] used morphology for small object rejection and hole filling on a binary image of the cervical smear scene, without explaining the important step of how the binary image was initially obtained.