Integrating Language and Vision Models for Real-Time Assistance for the Visually Impaired
摘要
Human perception relies heavily on vision, which enables people to engage with their environment in meaningful ways. Lack of visual perception makes it extremely difficult for people with visual impairments to navigate and carry out daily duties on their own. For the provision of context-sensitive, sound-related assistance to blind users, in this work, a real-time assistive system is proposed using state-of-the-art language and vision models. Real-time and precise assistance is provided through the system using a modular approach combining efficient item recognition and a language model for a rich scene description. Real-time object detection, visual content semantics, and assistive technology user interface are the primary components delivering an end-to-end assistance system. By addressing the urgent necessity for accessibility, this method provides improved independence and accessibility to the blind group of people.