Constructing Prototype-Based Granular Fuzzy Rules for Scene Classification on Mobile Devices

In this paper, we propose a method for scene classification by distilling knowledge from the pre-trained object detector into prototype-based granular fuzzy rules. The problem of classifying real-world scenes on mobile devices remains unresolved due to the complex contextual relationships between objects. The scene structure for a particular class is specified in the form of prototypes. The classification process is then inverted by reconstructing visual relationships based on the proximity of textual descriptions. Visual phrases are modelled by relation matrices “granules (objects) - granular structures - prototypes”, and the problem of visual relationships detection is reduced to solving the System of Fuzzy Relational Equations (SFRE). The solution set of the SFRE is translated using the set of minimum length rules or the unique maximum length rule. Visual relationships are restored by training the prototype-based fuzzy relational neural network. The visual phrase is formed by repeated runs of the training cycles to produce a set of candidate boxes covering granular structures. The max-min network is retrained to add new granular structures to the visual phrase; the dual min-max network is retrained to expand the bounding box for a particular granular structure. Fragments of a visual phrase related to one prototype in a given image area are enclosed into the union bounding box. Visual relationships detection by solving the SFRE allows to simplify the reconstruction of object-stacking scenes for real-time mobile applications. The accuracy of reconstruction depends on the set of prototypes configured by the user.