ACM Human Factors in Computing System (CHI) 2023.

Piano learning applications in Mixed Reality (MR) are a promising substitute for physical instruction when a piano teacher is absent. Existing piano learning applications that use visual indicators to highlight the notes to be played on the keyboard or employ video projections of a pianist provide minimal guidance on how the learner should execute hand movements to develop their technique in performance and prevent injuries. To address this gap, we developed an immersive first-person piano learning experience that uses a library of targeted visualizations of the teacher’s hands and 3D traces of hand movements in MR.