Bond: A Screen That “Sees” Emotions

Exploring the possibilities of emotional connections between users through movement-tracking multi-user public installations.

Individual Work | Undergraduate Capstone (September 2020 - May 2021)

Bond is an interactive installation that aims to explore the possibilities of humans forming an emotional connection with each other through movement-tracking encouraging physical encounters. It takes the form of multiple particle systems in the 3D space, where the expansion of particles represents the stimulation of Oxycontin, the brain chemical released in response to social bonding. Bond pushes the boundaries of the role technology has been playing in human interactions. From encouraging competitive to collaborative relationships, from 2D to 3D visual responses, previous research and multiple user-testing experiments have set a valid foundation for the development of this project. This is just a start to opening up many more opportunities.

“I saw someone walking towards me in the hallway.”
“We both stopped and looked at ourselves on the big screen.”
“That’s how it started.”

| Background

Movement-tracking is a technology that is used to understand human body movements. When we think of an interactive installation that utilizes this technology, oftentimes people are controlling animated avatars by standing in front of a screen.

One of my previous projects, DILTER, is a machine-learning-driven interface that introduces the fundamental concepts of movement-tracking technology to all-aged groups, especially children.

With the rapid development in technology and human interactions being digitized, the emotional connections between people are being “eaten up” by screens. We receive instructions on how to interact and form a connection with others through our virtual presence. As we stand in the phase of transitioning to the age of intelligence, perhaps we could experiment with ways to take advantage of digital tools to enhance our emotional connections with something we have always been familiar with, our bodies.

Through the relationship being set with users, whether it’s competitors or coordinators, they will have some form of emotional connection between them. So as these all serve as an inspiration, this project focuses on discussing the question “Through controlling an interactive avatar, how can body movement be used to create or increase an emotional connection between people”?

| Research

Conceptual Research

Research Question:

“Through controlling an interactive avatar, how can body movement be used to create or increase an emotional connection between people?”

Users can swipe their arms to shoot particles at each other as a weapon. There are no winners but the users will form a relationship as competitors as they play. This is a piece that takes the advantage of creating an emotional connection by setting relevance to the avatar. The users, without any communication, will place themselves in the role of the avatar, and therefore create an emotional connection.

As the participants engage in skin-to-skin contact and gestures, flowers bloom on the 3D-animated cherry blossom tree. It’s a strong piece that conveys a clear message about the importance of physical contact. It serves as a perfect example of how movement can be combined with screens to increase the emotional connection between users.

Experimental Research

Research Question 01:

“What elements would influence triggering an emotional connection in a multi-user interactive experience?”

Two prototype experiences were designed to help this research.

Prototype Experience 01 aims to recreate the mechanisms of how the body is being drawn in “Collaborative Control”. By inviting participants who have weak interpersonal ties, the prototype experience seeks to explore not only the participants’ interaction in a shared experience but also what are some environmental elements that will be influential to the experience. The prototype starts with a before-experience survey, then users engage with the installation, and finishes with an after-experience survey as users react to their experience videotaping footage.

This prototype was developed using PoseNet for the movement-tracking part.

Capstone Prototype Experiment - Connected Body

Research Question 02:

“How does the multi-user experience set in a public space affect users forming an emotional connection?”

Prototype Experience 02 is an upgrade from Prototype Experience 01. Drawing from user interview insights of the previous experiment, I’ve transformed it into a goal-oriented experience. The users would start by freely exploring. During the exploration, a hidden function would be triggered if the users perform a specific movement together, which is if their hands are close to each other.

In this experiment, to discuss the matter of whether the experience being placed in a private setting or a public setting would affect the outcome, I moved this prototype to the hallway. I’ve set a camera on the side to document people’s realistic reactions to running into this installation in the hallway.

The last experiment was developed by PoseNet, which didn’t work well. To explore the best technology to support this research, this one was developed using Microsoft Kinect V2 for the movement-tracking part.

Capstone Prototype Experiment - Body Paint

| Develop

Narrative

Tech Development

| Final Work

Exhibition - Future Proof: IMA/B Capstone Show 2021, Shanghai, China

back to Design Research