Publication

🏆 Secrets of a smile: Can internal experiences be derived from external expressions?

Mirella Hladký, Jozef Hladký, Tanja Schneeberger, Patrick Gebhard, Tomohiro Nagashima, Philipp Müller

ACII2026 (Best Student Paper Award) · 2026

@inproceedings{hladký2026,
  title     = {🏆 Secrets of a smile: Can internal experiences be derived from external expressions?},
  author    = {Mirella Hladký and Jozef Hladký and Tanja Schneeberger and Patrick Gebhard and Tomohiro Nagashima and Philipp Müller},
  booktitle = {ACII2026 (Best Student Paper Award)},
  year      = {2026},
}

Abstract

Computational emotion recognition traditionally relies on observable signals, but this approach fails in emotionally challenging social situations, where individuals actively regulate their expressions, e.g., through smiles and carefully chosen words. This regulation creates a critical mismatch between internal experience and external expressions, leading systems that typically associate positive expressions with positive emotions to misinterpret these signals. In this paper, we study to what extent self-reported internal functions of smiles can be decoded from multimodal observable behavior. To this end, we extend an existing corpus of smiles in shame-eliciting situations by adding comprehensive verbal annotations, creating a unique multimodal resource of 172 smile instances from 27 participants that enables comprehensive analysis of social and emotional signals in challenging contexts. Analyses using both computational models and human observers failed to reliably interpret smiles, e.g., whether a smile reflects or masks an internal emotion or serves self-regulatory, relationship-maintaining, or relationship compromising functions. Our results suggest fundamental limitations of interpreting internal states based purely on observable expressions, and that a reliable interpretation of complex internal states may require intrapersonal information from introspective self-reports. This work contributes novel multimodal annotations, available as an open research resource, and provides evidence that between-person variability limits universal approaches to interpreting smile functions.