In this paper, we have presented a new approach to recognizing hand signs. In our approach, motion understanding (the hand movement) is tightly coupled with spatial recognition (hand shape). To achieve a high applicability and adaptability to various conditions, we do not impose priori features that the system must use, but rather the system automatically derives features from images during learning using the principle of multiclass, multidimensional discriminant analysis. The recursive partition tree guarantees that the best subset features are selected to distinguish the specific subset classes. This approach combined with our previous work on the hand segmentation forms a new framework which addresses three key aspects of the hand sign interpretation, that is, the hand shape, the location, and the movement. The framework has been tested to recognize 28 different hand signs. The experimental results have shown that the system achieved a 93% recognition rate. It is shown that our approach provides better performance than the nearest neighbor classification in the eigen subspace.