While sign language recognition has often been tackled in the literature, solutions have either used infrared sensors or data gloves to extract hand skeleton information, or had to rely on large image datasets for visual recognition. Most approaches have been limited to recognizing letter signs, which are actually rarely used by signers. In this paper we lay the foundation for a novel approach for visual recognition of American Sign Language (ASL) signs, that relies on small image sets, and can be easily extended to cover many of the commonly used signs. Our solution is built using a reliable set of hand landmarks (extracted using Google’s MediaPipe hand tracker), and implements the first of the five ASL parameters, handshape. We show that we can successfully classify the handshape in signs where only a very small number of training images might be available. Future work will tackle additional ASL parameters, further improving recognition performance.