Multimodal interaction in collaborative virtual environments

Goto, T. and Escher, M. and Zanardi, C. and Magnenat-Thalmann, N.

Abstract: Human interfaces for computer graphics systems are now evolving towards a total multi-modal approach. Information gathered using visual, audio and motion capture systems are now becoming increasingly important within user-controlled virtual environments. This paper discusses real-time interaction through the visual analysis of human face feature. The underlying approach to recognize and analyze the facial movements of a real performance is described in detail. The output of the program is directly compatible with MPEG-4 standard parameters and therefore enhances the ability to use the available data in any other MPEG-4 compatible application. The real-time facial analysis system gives the user the ability to control the graphics system by means of facial expressions. This is used primarily with real-time facial animation systems, where the synthetic actor reproduces the animator’s expression. The MPEG-4 standard mainly focuses on networking capabilities and it therefore offers interesting possibilities for teleconferencing, as the requirements for the network bandwidth are quite low.

  booktitle = {International Conference of Image Processing (ICIP '99)},
  author = {Goto, T. and Escher, M. and Zanardi, C. and Magnenat-Thalmann, N.},
  title = {Multimodal interaction in collaborative virtual environments},
  month = oct,
  year = {1999},
  topic = {Facial Animation}