cici
A mobile app meant to provide live closed-captioning through realtime speech-to-text for students in lecture halls.
Role: UX Design Lead
Team: 2 developers, 2 designers
Problem Space
Students can have difficulty listening to lectures for a variety of different reasons - it might be hard to pay attention with other students on their phones or laptops, they could be uninterested in the lecture material, or the acoustics of the room might make it hard for students to hear what the lecturer is saying. A common underlying issue, however, appeared to be that students had difficulty listening to and therefore understanding lecture material, which made the class just that much harder or unengaging.
Design Question
How can we design a mobile application that can help students better understand lectures via a visual aide?
Research
For our research, we conducted five, quick interviews with other hackathon attendees (all of whom were current UW students). Our key findings from these interviews were, as follows:
Students found it hard to follow lecturers who talked too fast/slow
Students disliked that sometimes the lecturer’s voice was too quiet to hear
Students noted it was difficult to follow lecturer’s topic at times, especially if they went off-tangent
Ideation
Using the information we gathered from our research, our group began exploring what we could create. We knew that the Google Speech-to-Text API and the Google Translate APIs were ways we could implement our product; we also knew that we wanted to create a mobile application, since this would allow our users - college students - to easily prop their phone on their desk and pull up the app as a visual aide to what their lecturer would be saying. Based on these ideas, we came up with the following flow (left).
This led us to sketching low-fidelity wireframes on paper and sticky notes, to get a basic gist of what we wanted the high-fidelity mockups to look like, as well as how we wanted each screen to interact with the others.
Mockups
Taking inspiration from existing transcription applications like Otter and Steno, we created several high-fidelity mockup screens meant to visualize the following functions:
View and select previously recorded (and transcribed) sessions
Edit transcription errors by tapping by the word to lead to a blinking text cursor and the keyboard, like a text editor
Hard press words to select either bookmark or define, in which:
"Bookmark" would save keywords to review for later
"Define" would bring up a definition of the word in English, as well as another language that the user could select beforehand in settings
Second Iteration
Initially presenting this to mentors present at the hackathon, however, presented an important critique: the app resembled more of a dictation app, rather than an actual closed-captioning application that could provide these transcriptions effectively in real time. What were the visual cues that made closed captions so easy to read? How could users know when the app was recording vs when it wasn’t? How could we adjust the visual aesthetic of the current live-recording screen to better reflect this? With this in mind, we created two new screens meant to replace the original recording screen in a way that was more adherent to traditional closed-captioning formats:
Conclusion
I really appreciated the critiques we received that helped our group understand where we could improve on making the app feel more like “closed captioning anywhere”. By switching to a landscape orientation view and creating clear contrast between text and the background, it placed more focus on the text than the waveforms and recording features added onto our initial screen that made our product feel more like a dictation app than a true live-transcription application. With this iteration, however, we realized that even more improvements could be made to still relate it visually to the other screens we designed. Perhaps color or typographical elements could be tweaked to make the recording screens more familiar, or icons like the live indicator or record button could be given some additional - but not excessive - visual flair.
In using Android Studios to develop our mobile application, we found that often times the starter code did not work exactly the way we wanted it to, so it was definitely difficult to look for documentation and troubleshoot every error. Below is a recording of our prototype in action!