Charles Chung

What's the issue

People frequently use mobile phone and constantly need to stay in touch with others. With the existing method, sometimes they are unable to reply to an instant messaging (IM) right in time. For instance, when people are driving, cooking, or walking in a hurry, they may find it difficult to take out their phone or even switch visual attention toward the phone screen. A more natural method to receive and respond to IM is needed in this situation.

How I address the issue

A Voice User Interface (VUI) could be a suitable solution. While existing products provide either a read out loud service for an incoming notification or a speech to text text composing function, there isn't a service which combines the receiving and responding procedurals altogether to provide the user a Hands-Free, Eyes-Free interaction to the mobile phone.

In order to make a prototype out of this idea, I utilize NotificationListenerService on Android to listen to the notification sent to the phone. Then, a TextToSpeech library is called to read the incoming message out loud to the user. At the same time, the PendingIntent and RemoteInputs of the incoming notification are collected. These are the elements needed to construct the reply message later. The next step is to prompt the speech input. I implement a RecognizerIntent to receive speech result and transfer the audio into texts. The result then is added to the RemoteInputs and sent to the message sender via the PendingIntent, now that the text response is sent and so the procedure finishes.

What I learn from this

Form this project, I gained more programming skill for Android development. Especially, I learned how to utilize Service to have an app run in the background. Moreover, I gained a more mature understanding of how each function in the Android engine communicates with each other. This experience helped me to become a more solid programmer and developer.

Furthermore, the process of developing this idea and discussing with prof. Stanley helped me a lot to improve my ability to define and state the research question clearly. This is a valuable training experience for me.