The chapter considers the problem of automatic recognition of sign images for computer vision systems. It is shown that the main purpose of such systems is to extract from images information useful for further use in various applications. In particular, this chapter discusses issues related to sign language recognition and identifies potential negative factors affecting the accuracy of sign recognition. It is noted that for computer vision, hands are a difficult object to perceive during sign language communication. The accuracy and efficiency of sign language recognition algorithms are also affected by low image quality, variety of poses and orientations, non-uniform backgrounds and others. The preparation of initial data, creation of training and test dataset is described. For this purpose, a custom data set is created, and integration and preparation of external data is performed. An example of program code and generated own images is presented. A research of popular neural network methods for recognizing sign language in the presence of negative factors is performed and an evaluation of their effectiveness in the presence of negative factors is obtained. It is noted that knowledge of negative factors and their effect on the recognition quality will help to improve the efficiency of the recognition system. The recommendations for improving the quality of sign recognition are given.