Dr.-Ing. Christian Hacker   Dr.-Ing. Christian Hacker

-Assessment of non-native speakers
-Children's Speech
-Focus of Attention
-User States
-Speech Recognition


Multimodal classification of the focus of attention


In the German SmartWeb project, the user was interacting with the web via a Smartphone to get information on, for example, points of interest. To overcome the tedious use of devices such as push-to-talk, but still to be able to tell whether the user is addressing the system or talking to herself or to a third person, we developed a module that monitors speech and video in parallel. Our database has been recorded in a real-life setting, indoors as well as outdoors, with unfavourable acoustic and light conditions. With acoustic features, we classify up to 4 different types of addressing (talking to the system: On-Talk, reading from the display: Read Off- Talk, paraphrasing information presented on the screen: Paraphrasing Off-Talk, talking to a third person or to oneself: Spontaneous Off-Talk). With the camera of the Smartphone, we record the user's face and decide whether he is looking onto the phone or somewhere else. We use three different types of turn features based on classification scores of frame-based face detection and word-based analysis: 13 acoustic-prosodic features, 18 linguistic features, and 9 video features. The classification rate for acoustics only is up to 62 % for the four-class problem, and up to 77 % for the most important two-class problem "user is focussing on interaction with the system or not". For video only, it is 45 % and 71 %, respectively. By combining the two modalities, and using linguistic information in addition, classification performance for the two-class problem rises up to 85 %.

Investigated problem

Classification, whether the user of a smartphone communicates with the system by speech input or not.
The image shows the user from the perspective of a smartphone cam:

Classification of On-View/Off-View, ROT (read off-talk), POT (paraphrasing off-talk), SOT (spontaneous off-talk), and NOT (no off-talk = on-talk):


For automatic classification of the focus of attention we used Haar-Wavelets/Adaboost for On-View/Off-View detection and prosodic features for On-Talk/ROT/POT/SOT classification. Linguistic features (POS = part of speech, e.g. content word follows) are used in the third classification task. Fusion of the modalities was based on meta features describing each sub-system in low dimensional featre space:


The demonstrator allows step by step speech recording, recognition, and classification (left). In the right part, whe word based recognition results are shown together with On-Talk scores. In the middle the sentence based scores for On-Talk/Off-Talk, On-View/Off-View, and after fusion (On-Focus/Off-Focus) are shown.

Own Publications

all publications

Batliner, Anton; Hacker, Christian; Nöth, Elmar To Talk or not to Talk with a Computer - Taking into Account the User's Focus of Attention Journal on Multimodal User Interfaces, vol. 2, no. 28, pp. 171-186, 2008

Batliner, Anton; Hacker, Christian; Kaiser, Moritz; Mögele, Hannes; Nöth, Elmar Taking into Account the User's Focus of Attention with the Help of Audio-Visual Information: Towards less Artificial Human-Machine-Communication In: Krahmer, Emiel; Swerts, Marc; Vroomen, Jean (Eds.) AVSP 2007 (International Conference on Auditory-Visual Speech Processing 2007 Hilvarenbeek 31.08.-03.09.2007) 2007, pp. 51-56
[download poster]

Hacker, Christian; Batliner, Anton; Nöth, Elmar Are You Looking at Me, are You Talking with Me -- Multimodal Classification of the Focus of Attention In: Sojka, P.; Kopecek, I.; Pala, K. (Eds.) Text, Speech and Dialogue. 9th International Conference, TSD 2006, Brno, Czech Republic, September 2006, Proceedings (9th International Conference, TSD 2006 Brno 11-15.9.2006) Berlin, Heidelberg : Springer 2006, pp. 581 -- 588 - ISBN 978-3-540-39090-9

Nöth, Elmar; Hacker, Christian; Batliner, Anton Does Multimodality Really Help? The Classification of Emotion and of On/Off-Focus in Multimodal Dialogues - Two Case Studies. In: Grgic, Mislav; Grgic, Sonja (Eds.) Proceedings Elmar-2007 (Elmar-2007 Zadar 12.-14.09.) Zadar : Croatian Society Electronics in Marine - ELMAR 2007, pp. 9-16 - ISBN 978-953-7044-05-3

Batliner, Anton; Hacker, Christian; Nöth, Elmar To Talk or not to Talk with a Computer: On-Talk vs. Off-Talk In: Fischer, Kerstin (Eds.) How People Talk to Computers, Robots, and Other Artificial Communication Partners (How People Talk to Computers, Robots, and Other Artificial Communication Partners Bremen April 21-23, 2006) 2006, pp. 79-100

Maier, Andreas; Hacker, Christian; Steidl, Stefan; Nöth, Elmar; Niemann, Heinrich
Robust Parallel Speech Recognition in Multiple Energy Bands In: Kropatsch, Walter G.; Sablatnig, Robert; Hanbury, Allan (Eds.) Pattern Recognition, 27th DAGM Synopsium, Vienna, Austria, August/September 2005, Proceedings (27th DAGM Synopsium, Vienna, Austria, August/September 2005 Wien 30. August - 2. September) Vol. 1 Berlin : Springer 2005, pp. 133-140 - ISBN 3-540-28703-5