Stepping stone towards goal-oriented Human-Computer Interaction.
Here, I highlight my research regarding the concept of Productive Engagement: it’s conception, modelling, and evaluation in real-world settings after iterative research over 3 years and with ~300 children. This work is situated in a robot mediated collaborative learning activity JUSThink [6] where we looked at student’s interaction with the activity, speech patterns, facial expressions, and gaze patterns. However, the concept and methodological pipeline is transferable to all other goal-oriented Human-Robot Interaction.
“Productive Engagement is the engagement that is conducive to ‘the goal of the interaction’”
[1] Nasir, J., Bruno, B., Chetouani, M. et al. What if Social Robots Look for Productive Engagement?. Int Journal of Social Robotics 14, 55–71 (2022). https://doi.org/10.1007/s12369-021-00766-w
In this work [1] we laid down the foundation of the concept of Productive Engagement and validated the hypothesis in a data driven manner using multimodal human behavior in educational HRI settings. Productive Engagement [1] is the engagement that is conducive to ‘the goal of the interaction’ which in this case is learning. Such modelling of engagement represents a perspective shift towards aligning it with the ultimate goal of the interaction which in this case is student learning and is the cornerstone of the design of any social educational robot.
[2] Nasir, J., Kothiyal, A., Bruno, B. et al. Many are the ways to learn: identifying multi-modal behavioral profiles of collaborative learning in constructivist activities. Intern. Journal of Computer-Supported Collaborative Learning 16, 485–523 (2021). https://doi.org/10.1007/s11412-021-09358-2
For the validation and to surface labels in data-driven manner, we proposed a forward-backward technique (summarized in [2, 3]) utilizing clustering which is transferable to other application areas. This technique caters for cases when more than one behavioral profiles are associated with productive engagement. With this technique, we were able to extract the behavioral profiles of students pertaining to productive and unproductive engagement in the context of collaborative learning.
[3] Nasir, J., Abderrahim, M., Kothiyal, A., and Dillenbourg, P. Temporal Pathways to Learning: How Learning Emerges in an Open-ended Collaborative Activity. Computers & Education: Artificial Intelligence Vol 3, (2022).
Since learning does not happen in a single moment but rather is a process, in [4], we generated temporal behavioral profiles of students pertaining to productive and unproductive engagement in the context of collaborative learning. These temporal profiles were generated making use of Hidden Markov Models.
[4] J. Nasir, B. Bruno, and P. Dillenbourg, Social Robots as Skilled Ignorant Peers for Supporting Learning, Frontiers in Robotics and AI, vol. 11, 2024, ISSN: 2296-9144. DOI: 10.3389/frobt.2024.1385780
Now that we have behavioral profiles indicative of productive engagement, we developed methods for real-time assessment of Productive Engagement in HCI settings [5, 6]. In the first method [5], we developed a two layered approach which makes use of a productive engagement score and then the behavioral profiles of the students to classify them. This classification is followed by an intervention from the robot if the students are not engaged productively.
[5] J. Nasir, M. Abderrahim, B. Bruno and P. Dillenbourg, An HMM-based Real-time Intervention Methodology for a Social Robot Supporting Learning, 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Busan, Korea, Republic of, 2023, pp. 2204-2211, doi: 10.1109/RO-MAN57019.2023.10309430.
In the second method [6], we developed an HMM based real time methodology that fuses information from all modalities early on and tracks an unproductive engagement score followed by interventions whenever the score crosses a threshold.
Jauwairia Nasir 