Architectures and Models for Interaction Group

Research topic of the group AMI: Image and Interaction

E. Frenoux, F. Abdin, D. Béroule, T. Bouchara, H. Ding, S. Fdili Alaoui, A. Hasasneh, C. Jacquemin, A. Osorio
Image theme regroups several research interests concerning either augmented reality or computer vision or medical imaging. Each of them aims at developing new technologies for an automatic treatment of digital images and Human-visual environment interaction improvement.

Advanced Medical Imaging.

As surgical technologies are more and more laparoscopy-oriented, surgical punctures became a wide-spread technique for diagnosis and follow-up. As the clinician has to rely on the images to make therapeutic decision and plan surgical intervention, one has to feel completely confident about medical images and to have user-friendly data representations. Our research aims at providing the practitioner tools allowing the best action possible. Since 1990, a computerized interactive system called PTM3D has been developed and enriched. It is used online, for a help to diagnosis, evaluation, surgical planning and execution. The system deals with DICOM images coming from any medical image modality, allowing volumetric segmentation and 3D visualization of anatomical structures and lesions. The research interests can be divided in three poles:

• Interactive semi-automatic segmentation of structures and lesions in a 3D environment for surgical planning
• Computation of the projection of the segmentation onto the real body’s envelope, inside the operating room, to allow the surgeon to see through the patient’s body (Illustration 1).
• Real-time follow-up of surgical tools, using a webcam, to validate the operation course with respect to the initial planning.

Clinical applications are driven in collaboration with several French, Spanish, English and American hospitals, and concern kidney surgery, gastroplasty and hepatectomy. The main characteristic of the system is that the surgical protocol isn’t modified by the use of our tools, as it only needs visual projections.

Projection computation for online surgery helping, a screen capture: the orientation of the projector is computed from the puncture path (displayed right, with CT images for control) and the patient’s body inclination (displayed left, with the projector and the entrance point for the puncture).

Image Processing for Augmented Reality and for Robotic Vision

1. Image processing for Augmented Reality. For Spatial Augmented Reality (i.e. using video-projection to overlay physical space with visual digital data), it is necessary to calibrate the image projected onto the physical world, in order to find its optimal position. This point is developed through an external collaboration. In addition to the core calibration issues, Spatial Augmented Reality raises many image processing challenges, such as:

• Computing projection masks so that visual augmentation can be limited to subsets of a real scene (e.g. spectators shadows)
• Real-time image transformation to re-project it onto the scene, after modification (e.g. contour delineation)
• Human-scene interaction management.

All the algorithms are implemented on the Graphic Processing Unit in order to optimize processing time and make them compatible with real-time interaction.
Concerning calibration and real-time images correction, a research collaboration has been developed with the IEF lab (team ACCIS). This work should allow rebuild the physical world geometry and use it for computing image correction on plane parts of the physical scene.
Our researches concerning Spatial Augmented Reality are used in various projects: for built heritage  augmentation, for interactive installations in public spaces, and more generally, for some of the art/science applications described in VIDA transversal theme.
Three PhD are currently studying applications and extensions of image processing for Augmented Reality: Hui Ding is studying audio-graphic scenes descriptions and rendering in the framework of the ANR Topophonie project. Her results can be applied to audio and visual augmentations of physical scenes. Tiffanie Bouchara is developing comparative analysis methods for visual and auditory perceptions in audio-graphic scenes. Sarah Fdili Alaoui PhD proposes new perspectives for gesture interaction using the whole body and motion analysis in collaboration with IRCAM.

PICRI project “Toute la lumiÚre sur l’ombre”, in collaboration with didascalie.net and “L’ange Carasuelo” companies and with Taverny’s library “Les Temps Modernes”: real-time image processing for video-projection inclusion in shadows or in a silhouette captures.

2. Image Processing and Robotic Vision. This theme is the object of collaboration with members of the CPU group of the LIMSI. The PhD of Ahmad Hasasneh concerns the development of machine learning methods for semantic place recognition and robot localization.