This book constitutes the thoroughly refereed post-workshop proceedings of the Fourth IAPR TC9 Workshop on Pattern Recognition of Social Signals in Human-Computer-Interaction, MPRSS 2016, held in Cancun, Mexico, in December 2016.The 13 revised papers presented focus on pattern recognition, machine learning and information fusion methods with applications in social signal processing, including multimodal emotion recognition, user identification, and recognition of human activities.