About Pianow Project

Learning to play the piano is a difficult process that requires constant and rigorous practice. While it is believed that 65% percent of human beings are visual learners, traditional methods of teaching and learning to play the piano have not been focusing on exploiting the potential of human vision. For example, young children, when learning to play the piano, are very often not allowed to see the names of the keys they press but are instead required to picture the keys in their minds. Meanwhile, in helping new learners picking up the instrument faster, the idea of using falling notes and colored keys as visual hints has proven to be highly promising by many applications. In this project, this idea will be brought into Mixed Reality, a combination that is expected to improve the learning experience, as learners can follow the tutorials without looking away from their own physical piano keyboard. The new application, named Pianow, aims to offer beginners a new way to practice playing the piano by themselves, which is easy, fun, and interactive. Pianow is developed for the HoloLens. It can effectively align virtual assets to the physical piano keyboard and display interactive piano tutorials consisting of falling notes and colored keys.

There are two major objectives to be achieved in this project:

  1. Pianow is reasonably accurate and effective in aligning its virtual content with a standard 88-key piano keyboard.

  2. Pianow offers at least 2 working tutorials with intuitive user interface and good user experience.

Watch Demo Video

Methodology

Alignment Methods

In this project, 3 alignment methods have been attempted.

  1. Manual Alignment

    A realistic 3D model of the piano keyboard is prepared. The user can drag and drop the model to align it with the physical piano keyboard. The implementation of this method is adapted from the HoloToolkit Collection.

  2. Keyboard Recognition with OpenCV

    A C++ OpenCV plugin is written to identify the piano keyboard's position and orientation from each frame captured by the HoloLens' locatable camera. The output pixel coordinates are returned to the main C# program and projected to the 3D world coordinate system to determine where the virtual contents should be generated.

  3. Marker Detection with Vuforia

    A marker is placed at a specific position relative to the piano keyboard. Upon detecting the image target, Vuforia performs feature-based pose estimation and generates the attached content based on the pose information of the image target detected.

Tutorial Content & UI

The tutorial is prepared in the form of a video, which is rendered into a video texture in Unity. This texture is then mapped to an L-shaped model shown in front of the user.

Some basic UI elements are adapted from the HoloToolkit Collection, but all UI event handlers are implemented from scratch.

Usability Testing

Subjects: 10 university students, with or without musical background

Setup: Camera recording

Procedure: Learning gestures → Testing alignment → Testing tutorials → Survey

Evaluation: Survey, observation, video recording analysis

Progress & Result

Alignment Methods

Generally speaking, the "Marker Detection with Vuforia" method currently offers the best alignment results. However, in view to the future improvements of the HoloLens, the "Keyboard Recognition with OpenCV" method is more prominent and is expected to offer better user experience.

  1. Manual Alignment

    Manual Alignment method is flexible as it gives the user the most control over the alignment process. The user can perform more precise adjustments of the hologram’s position according to his/her own judgement. However, this also means that the method requires more effort from the user compared to other methods. Also, it depends on the assumption that the user is familiar with using the HoloLens and has adequate skill to accurately align a hologram with real-world objects.

  2. Keyboard Recognition with OpenCV

    Keyboard Recognition with OpenCV method provides a better recognition result when the user faces the piano keyboard upfront, and the locatable camera can capture the entire keyboard. This method depends largely on the spatial understanding capability of the HoloLens because the spatial mapping mesh is used to determine the center of mass and orientation of the piano keyboard in world coordinates. Generally, the position and orientation of the hologram becomes more accurate as the HoloLens has more time to scan the environment and obtain a better spatial mapping mesh.

  3. Marker Detection with Vuforia

    Marker detection with Vuforia is a reliable method as the user can carry it out without having to move around the space too much. It also produces reliable and consistent results as the marker tracking mechanism provided by Vuforia is robust. Furthermore, this method does not depend too much on the capabilities of the HoloLens such as field of view or spatial map-ping mesh, so it effectively eliminates most shortcomings of this device.

Usability Testing

"Marker Detection with Vuforia" method is generally regarded by the test users as the best solution among the three thanks to its simplicity and accuracy. Around 70% of the test users think that the colored keys and falling notes may help them learn to play the piano. Subjects with musical background tend to have better hand placement compared to subjects that are absolute beginners.

Project Plan - 01 Oct 2017

This Project Plan provides details of the project’s motivations, scope and objectives, the methodology to be used as well as how the project will be carried out.

Download

Interim Report - 29 Jan 2018

This interim report discusses the significance of Pianow, solutions to some technical challenges, the current progress and the future plan of the project.

Download

Final Report - 15 Apr 2018

This final report discusses the significance of Pianow, solutions to some technical challenges, the current progress and the future direction of the project.

Download 1

Download 2

Dr Dirk Schnieders Supervisor

Lecturer at the Department of Computer Science, The University of Hong Kong

  • (+852) 2857 8440
  • sdirk@cs.hku.hk

Bui Thanh Tung Student

Fourth year student majoring in Computer Science at the University of Hong Kong

  • (+852) 5441 2529
  • ethanbui@connect.hku.hk

Tsai Yi-Ting Student

Fourth year student majoring in Computer Science at the University of Hong Kong

  • (+852) 6226 2793
  • u3512452@connect.hku.hk
Top