Two CS Students Who Love Italy Build Virtual Reality Language Program

“We came up with this idea to mimic the experience we were having being totally immersed in the Italian language,” Bridget Went ’17 said. “Being in an environment where the language is spoken helps our ability to synthesize components of the language and improves our fluency.”

At the Università di Bologna, Went and Clarissa Hunnewell ’17 partnered with two Italian peers to begin building a virtual reality tool for learning Italian. They made enough progress with the program, originally a desktop platform they called Esplora, that they decided to continue working on it after returning to Bowdoin.

Advised by both Professor of Romance Languages and Literatures Arielle Saiber and Associate Professor of Computer Science Stephen Majercik, the two students spent the fall semester learning how to use a virtual reality software called Unity. They also studied foreign language teaching concepts and techniques. Saiber praised the project for being a “great example of interdisciplinary work, digital humanities, and collaboration across departments,” and the “impressive vision and talent” of the students.

Went noted that she and Hunnewell have “a rare combo” of interests that led them to the unique project. “We’re both computer science and math majors, and Italian minors,” she said. “Seeing the potential for really creative applications of our skill-set is cool and it is a great interdisciplinary project that combines two of our three main fields of interest.”

Stephen Houser, Bowdoin’s director of Academic Technology and Consulting, helped the students by acquiring an oculus, which outfits users with a headset and hand sensors for a vivid, interactive 3D experience.

In the spring, Hunnewell and Went began making the virtual program. By the time they graduated in May, the students had created one main interactive scene set in an open urban square. Went and Hunnewell also began building two more scenes, one a “dictionary room” and another a classroom, which are connected to the piazza.Future scenes — like a bar or a restaurant — will also be accessible from the piazza.

Visiting Venice: Using Esplora

When you first log into Esplora, you’re prompted to give your level of fluency in Italian, and what follows is tailored to suit beginners, intermediate or advanced speakers of Italian.

The first screen opens onto the famous Piazza San Marco, and if you walk toward the church, bells start ringing. An avatar friend, Angela, meets you (her voice was voiced by this year’s Italian teaching fellow, Angela Lavecchia from Naples, Italy). She helps guide users through the experience.

After exchanging greetings with Angela, you can wander around the piazza, clicking on objects to see what they are called. Or, if you’re using the headset of an oculus, you can simply gaze at an object for a few seconds and the word and its pronunciation appear above it. Users are encouraged to collect new words into their virtual dictionary, which is a room you can visit to test your vocabulary recall.

One of the activities in the piazza is a scavenger hunt, which asks users to find several things, for instance, una statua, una panca, or la spazzatura (statue, bench, or trash). Another activity offers cultural and historical information that is relevant to the piazza, and users then answer questions about this topic in Italian.

A third activity has users asking either Angela or a stranger to take a picture of you in front of the church. Angela will correct you if your grammar is incorrect.

Esplora’s Future

Though Hunnewell and Went have jobs next year, they said they’d love to keep working on Esplora on the side. Went will be a software engineer at the San Francisco-based Okta, an “identity cloud company,” as she describes it, and Hunnewell will work at EverQuote, in Cambridge, Mass., which is a start-up insurance company.

Hunnewell and Went said that they’d like to keep building scenes in Esplora, eventually adding voice recognition and a multi-player platform. “This would enable American and Italian students to interact in a kind of pen-pal way and exchange information about themselves and contemporary culture,” Went said.

At the moment, programs that bring virtual reality into schools are somewhat rare, and the two know they’re on the edge of a burgeoning field. “The field is evolving so quickly,” Went said. “Now there are so many organizations out there beginning to do what we want to do with language instruction.”

Saiber said she is excited by the possibilities of Esplora and other programs likes it. “I would love to see continued collaboration between the faculty and students of computer science and language and literature departments,” she said. “There is so much we can offer one another in terms of thinking about the structure of language, human and machine learning, and human and machine interface.”