Abstract
This study is the result of a joint effort of three research groups of the CHM Department in the investigation of a cognitive issue through the development and implementation of a general purpose VR environment. The psychological aspects of the study deal with mechanisms involved in spatial cognition, in particular to determine how the verbal description of an environment or the active exploration of that environment affects the elaboration of a mental spatial representation. Another issue is to investigate the role of vision by assessing whether participants without vision (blind from birth, late blind or blindfolded sighted individuals) benefit from these two learning modalities. This work is the continuation of a series of experiments led in the framework of a thesis in Cognitive Science (funded by a grant from the French Ministry of Research). The study aimed at shedding light on the effect of visual deprivation on the capacity to mentally represent spatial configurations. The experiments were conducted with blind persons as well as with sighted persons (eyes opened in one group, blindfolded in another one).
Mental Imagery Tasks
A spatial configuration was designed in the form of a disk, 50 cm in diameter, placed upright in front of the participants. The locations of six objects on the periphery were learned by the participants through one of two modalities. The first one was a verbal description of the configuration while the second involved haptic exploration of the disk. The task consisted for the participants to create a mental representation of a spatial configuration, and then to compare distances between the objects situated on its periphery. The results showed that whatever the type of visual deprivation experienced by the participants and the learning modality, all the participants were able to create a mental representation of the configuration which preserved the metric relations between the objects. To achieve these same level of performance, blind people needed more time than the sighted, whether the latter were blindfolded or not during the experiment.
Two mental scanning experiments were conducted with the same populations, according to the same learning modalities. The task consisted in mentally imagining a spot moving between two objects and responding when the trajectory was completed. For all experimental groups and for both modalities, a correlation between response times and scanned distances was obtained. This result confirms the outcomes of previous mental scanning studies.
Immersive Audio Interface
While there was a significant difference in response times between blind and sighted participants, it was evident that this was in some part due to the nature of the task. We examined the hypothesis that people blind from birth, who are not expected to generate visual images, are nevertheless proficient at using mental simulation of trajectories, whereas sighted would perform better if they have the possibility to generate a visual mental image. This difference could explain the difference in the processing times needed by blind people in contrast to the sighted, and could explain the tendency for blind people's response times to be shorter after haptic exploration of the configuration. To test this hypothesis, a new experimental system was designed. A large-scale immersive audio virtual environment was created in which participants could explore and interact with virtual sound objects located within a room. Two learning conditions were contrasted, one expected to induce a mental representation of an iconic nature, and the other generating a mental representation based on perception/action couplings. The comparison was intended to help identify which learning modality generates the most accurate representation of a spatial configuration, as well as to collect information on the perception of a world by blind and sighted people interacting with a three-dimensional audio environment.
The task performed by the participants was the manual reconstruction of the sound scene. Additional tasks were also included, involving the presentation of virtual scenes including metric changes followed by a task consisting in correcting the scenes. Finally, mental scanning and comparison of distances tests were conducted.
Through the combined efforts of two other research groups, an immersive interactive audio environment was developed. The details of the system are presented at the following links:
- An immersive virtual multimodal environment for cognitive studies
- Audio Augmented Reality (http://perso.limsi.fr/jacquemi/ami8/#AudioAR)
Similar to the haptic environment described earlier, an audio sound scene was created by placing a number of sonic objects along the periphery of a circle. The virtual circle was placed within a room in which the participants were able to move about. Domestic sounds where chosen which could be played continuously and still sound natural. The following figure presents the subjective view and the overhead view of the virtual scene. The numbered spheres indicate the audio objects while the head and arrow represent the current position of the participant’s head and hand/pointer which were tracked. The participants (blind or blindfolded) only experienced the spatial scene though a spatialized 3D audio rendering, the graphic output was only used by the test supervisor.
In the task of scene reconstruction, the preliminary results suggest that active exploration of an environment favors the absolute positioning of sound sources as compared to learning from a verbal description. The same improvement appears with respect to radial distance errors, but only for blindfolded participants. In addition, the data show that, whatever the learning modality, participants underestimate the circle diameter, except for the case of blindfolded, who clearly benefit from learning with perception/action coupling.
We also found that blind from birth participants made significantly more angular positioning errors than late blind or blindfolded ones, whatever the learning modality of the environment.
|
Correlation curve : distance vs. verbal (Obs. to VM ignored) 
Scanning mental results of blindfolded sighted
people after a verbal learning of the configuration. |
Correlation curve : distance vs. explo (Obs. to VM ignored) 
Scanning mental results of blindfolded sighted
people after an active exploration of the configuration. |
In the mental scanning task, the correlation between distances and scanning times (which is the typical signature of the analog character of the mental representation) is found after the two learning conditions, but with stronger correlation values and with shorter absolute times after learning the verbal description. Blindfolded sighted participants' responses attest for special difficulties after learning by active exploration.
Video Examples
Two video examples are provided to better understand the system in operation. Video clip (1) shows the acutal physical environment in which the virtual reality system was installed, and shows a participant during the course of the exploratory phase of the experiement. Video clip (2) shows the virtual environement during several phases of the experimental procedure. The virtual scene is shown from both the participants subjective view and also the overhead view. The virtual audio scene is also included and is rendered for the participant's position in the scene.
ExcerptExpeCam.mov (2.1MB) |
ExcerptExpeVR.mov (49MB) |
Conclusion and future work
Our results point out the value of investigating the role of perception/action coupling in mental processing, while illustrating also how VR is a powerful tool for creating complex and interactive experimental contexts. This research is currently being extended in the form of a doctoral thesis supported by the European Union in the framework of the STREP "Wayfinding" (in partnership with the Collège de France). The study is intended to investigate the role of head movements in the analysis of sound scenes.
Another interest of this study can be viewed in relation to applications employing the presentation of spatial information through non-visual modalities. Investigating navigation with and without vision is a prerequisite to building navigation systems for blind people who are exposed to 3D audio information .
References
Presentations
- Afonso, A., Katz, B. F.G., Blum, A. & Denis, M. “Spatial knowledge without vision in an auditory VR environment.” Proceedings of the XIVth meeting of the European Society for Cognitive Psychology, Leiden, the Netherlands, 31 août - 3 septembre 2005.
- Afonso, A., Katz, B. F.G., Blum, A., Jacquemin, C. & Denis, M. “A Study of Spatial Cognition in an Immersive Virtual Audio Environment: Comparing Blind and Blindfolded Individuals.” Proceedings of ICAD 05, 11th Meeting of the International Conference on Auditory Display, Limerick, Ireland, 6-9 juillet, 2005.
- Afonso, A., Katz, B.F.G., Blum, A. & Denis, M. “Mental imagery and the acquisition of spatial knowledge without vision: A study of blind and sighted people in an immersive audio virtual environment.” Xth European Workshop on Imagery and Cognition, St Andrews, Scotland. 28 – 30 juin 2005.
- Blum, A., Afonso, A., Katz, B.F.G., Jacquemin, C. “Expérimentation sur la Perception de l’Espace en Réalité Virtuelle Immersive Audio”. Workshop IHM’04. Namur, Belgique, 30 Août – 3 Septembre 2004 (Communication affichée)
- Jacquemin, C., Afonso, A., Blum, A., Katz, B.F.G. “Une Interface Utilisateur 3D Multimedia Audio et Graphisme : Langage, Navigateur et Expérimentations”. Workshop IHM’04. Namur, Belgique, 30 Août – 3 Septembre 2004 (Communication affichée)
- Jacquemin C., Afonso A., Blum A., de Laubier S., Denis M., Folch H., Genevois H., Katz B., Nugier S., and Schnell N. (2005). Design d'environnements multimodaux interactifs communicants. In Proceedings, Hypermedias Hypertexts, Products, Tools and Methods H2PTM'05, Paris, France.