Closed-Loop Control of Images Based on Electrocorticogram Decoding in Visual Semantic Space
摘要
Neural representations of visual perception are intentionally modulated by mental imagery and attention. In the present study, we hypothesized that visual images decoded from electrocorticograms (ECoGs) could be controlled to represent intentional meaning in a closed-loop condition. ECoGs were recorded while subjects watched videos containing natural scenes, while the annotations of the scenes in the videos were converted into vectors in the visual semantic space using a word-embedding model; decoders were trained to infer the semantic vector from the ECoG. By presenting images based on the inferred vectors in real time, four subjects successfully controlled the images to show an orally instructed meaning. Closed-loop control of inferred images revealed a novel interaction between visual perception and imagery.