vrSocial

A game in Virtual Reality to help young autistic adults with social interaction skills via non-verbal cues. Developed and designed for HTC Vive set

Role: Game Designer and Developer (Team of 3)
Location: University of California, Irvine
Duration: 8 months
Tools: HTC Vive, Unity, Minecraft, Photoshop, Paint.Net

Background

A research group in the Social and Technological Research Group at UCI were interested in developing an application in Virtual Reality to help young autistic adults with social interaction skills via non-verbal cues. Previously, the group had designed a curriculum on similar lines and apps based on that were developed. This project is an extension of those studies, but in Virtual Reality.

Challenges

I had never done game development before, and this project required me to do that in the Virtual Reality domain. Also, the equipment being used is the HTC Vive, which I hadn't used until this project kick started. I hadn't worked on projects for users with special needs such as autistic kids either. Naturally I had to organically modify many of my user-centric views as the project went along.

Previous Curriculum

A curriculum in the form of cards was developed to teach autistic children about social skills. The poster depicts the social skills that were the center of this research. Mobile applications based on these social skills were built. These included an app for 'Proximity' - teaching kids how much distance to maintain with whom - and 'Prosody' - teaching them about how to use their speaking volume and pitch while conversing.

User Needs

We did a field study to The Center For Learning Unlimited - a school for autistic kids - to interact with the students and teachers there to uncover user needs. These are the key findings:

Needs from a teacher's point of view:

Non-verbal cues are essential.
Don't give them the solution, guide them towards one.
Avoid directive language.
Replicate real life scenarios to pause, rewind and replay.
Anything related to robot building will be loved.

Needs from a student's point of view:

Majority of the curriculum cards got a thumbs up
Should be able to do anything (like walking on the roof)
Difficulty remembering names - want auto identification in VR
Multiplayer would be very helpful

We performed card sorting activities with kids at a few different places, to identify the essential skills from the curriculum that they thought would be helpful if included in the game. The results from the tests were pretty encouraging seeing that the skills that we were setting out to develop were in fact considered important by the kids and that we weren't headed down the wrong path.

Storyboarding

We brainstormed various scenarios and the corresponding virtual reality game stories. Here are few of the storyboarding sketches that cropped up during the discussions:

Initial Prototype

Initially, we started building mods in Minecraft as a first step towards building games in virtual reality. A mod named 'Vivecraft' makes tweaking and playing Minecraft in Virtual reality possible, and we decided to take advantage of that. We tried building shields and auras for a player as part of our game to teach the social skill of Proximity.

Shield for the player to protect the personal space

A smokey aura around the player for personal space

Moving on from Minecraft

We shifted from Minecraft to build our own game on Unity. Our initial Unity prototype was focused on Proximity. Three concentric circles (zones) followed each user. Red zone represents the personal space, Green zone is the social space while the Blue zone represents the public space. In this prototype randomly generated characters spawned in the same room as the user. And the user was given the freedom to expand or contract his/her personal space bubble. If the user's personal space crossed into the personal space of any of the character, visual and audio cues would help the user to learn to respect the personal space of others. Similarly if the user wanted no one to enter his/her space, he/she could create an enclosing shield around, so that no character could breach into the red zone.

Random characters with their red bubbles

Shield blocking a character from entering into the user's personal space

Random characters in the user's public space

Back to Sketching

Once we were comfortable with developing in Unity, we shifted our focus back to the game experience, and how the other social skills games could be integrated with Proximity. Going back to the drawing board after brainstorming for a few days, we got came up with some paper sketches to put our ideas out.

User Tests and Design iterations

After several brainstorming sessions, we decided to let go the shield, and adjustable personal space function. We then developed an online multiplayer version of the game. Also, we were able to build and integrate other social skills into the game; Turn-Taking to help them maintain a balance in conversations and not speak too less or too much, Volume-meter to give them an indication if the volume at which they speak is too high or too low, and Body-orientation to help them maintain proper body orientation while interacting with the other player. As seen in the pictures, we started with spawning the other player (connected remotely) as a Pikachu. The Proximity zone-circles at the bottom would affect the scores of the players depending on whether they maintained proper distances. Volume-meter at the beginning gave textual cues if the user went quiet or too loud.

1. The first multiplayer prototype with Pikachu avatar and textual cues

2. More visual cues, and a less distracting environment

3. Only visual cues, and a completely open space to interact in

After conducting the first user test at the CHOC hospital, we observed that the users were sure excited to experience VR for the very first time; the organizers at the hospital too agreed that VR could actually help. But seeing how little impact the textual cues and scores had on the users' behaviors, we then iterated the design to add a visual component along with each component and also reduce the graphical clutter in the environment.
With the turn taking bar now developed and integrated as well, we conducted another user test. The second user test had improved results, but the feedbacks were convincing enough to remove the textual completely, and giving an empty and open space for the user to interact in.

Final Prototype

After the third user testing, we observed that the visual cues were too subtle for the users to observe, infer and react in the short period they were in the virtual environment. So, moving we designed more obvious cues to help them maintaining proper proximity, got the volume-meter to react more unsubtle way. We also introduced a human avatar to represent the other player in the game to engage the user for short sessions. Lastly, we had a late addition of a 'Warning' sign if the user is about to cross into the other player's personal space. The prototype is currently undergoing usability testings at CHOC to validate its deployability.

Human avatar of the second player with all social skills' indicators

When the player stands out of the blue zone for too long, a reminder to stand closer is displayed

As soon as the player crosses the other player's red zone, he/she is reminded to step back

The player is dominating the other player in turn taking

A warning zone, to prevent the players from crossing into each other's personal zones

A warning message, to prevent the players from crossing into each other's personal zones

Key Takeaways

vrSocial has been a big learning curve for me as a UXer. There were a lot of firsts - first time I laid hands on a VR headset, first time I took up a game design/development project, my first time working with users with special needs, first time I was involved in some thorough user testings - and I am so glad they happened. Going on field study, brainstorming every other week about what next to do, conducting user tests, and going back to the board and iterating your designs, developing them, and repeating it, it all has been a great experience to be a part of, and most importantly seeing how your work might actually have an impact on people's lives just adds a whole other meaning to the task at hand!

Put yourselves in your users' shoes. Empathy will guide you towards designing the right solutions

Design is never complete. The iterations are never done. Design, listen, improve, repeat

Designing experiences in VR begs for different skillsets to be brought to the table