Keystone Today

A team of Franklin & Marshall College students combed through six hours of Zoom class footage to identify gestures exhibited by participating professors and students.

The results were overwhelming. Over the span of four recordings, the team identified 1,497 gestures.Save

“That’s 1,497 cases of communication where the person performing it may not even be aware they were doing so,” said sophomore Ayako Belka, a biology and psychology double major. 

Under the tutelage of Elena Cuffari, assistant professor of psychology & scientific philosophical studies of mind, Belka was joined by senior Daisy Zang and sophomore Wendy Huang. 

While gestures have been widely studied, the limitations of digital screen borders provide an unexpected twist.

“That’s rare in gesture scholarship because we just haven't had a lot of conditions or studies where the gesture is partly blocked,” Cuffari said. 

The research team shows some of the most commonly used gestures found in Zoom footage.The research team shows some of the most commonly used gestures found in Zoom footage. Image Credit: Deb GroveAyako Belka ’25 (left), Daisy Zang ’23, Professor Elena Cuffari, and Wendy Huang ’25 meet weekly to discuss their Zoom gesturing research.Ayako Belka ’25 (left), Daisy Zang ’23, Professor Elena Cuffari, and Wendy Huang ’25 meet weekly to discuss their Zoom gesturing research. Image Credit: Deb GroveElena Cuffari, assistant professor of psychology & scientific philosophical studies of mind, meets with students to discuss Zoom gesturing research.Elena Cuffari, assistant professor of psychology & scientific philosophical studies of mind, meets with students to discuss Zoom gesturing research. Image Credit: Deb Grove

“I also wanted to build up a corpus. This is a cool moment to have a bunch of data of people gesturing on Zoom. Yes, people meet on Zoom all the time, but it's not always so recorded and available and abundant. We had ninety-five hours of classes and we got to see professors’ and students’ behavior in a lot of detail in different class meetings,” Cuffari said.

Research was conducted by taking recorded classroom meetings from four different F&M professors during the 2020-2021 school year, and coding certain aspects of all observed gestures through a linguistic annotation system, EUDICO Linguistic Annotator (ELAN).

To decode the clutter, the team created their own coding template “so that we could focus on how much information the gestures are bringing to an interaction, as well as their spatial orientation within the Zoom frame,” Belka said.

Overall, the study showed that the most common type of gestures were ones that were only partially visible in the Zoom window, contradicting the team’s initial hypothesis that more meaningful gestures would have better visibility. 

To further determine the impact of these partial gestures, the research team is conducting a follow-up study that examines the effects of gesture visibility and attention span. 

"Gesturing is a form of communication that is so routine to us that we hardly notice it, yet it serves a huge role in daily interactions by communicating emotions and ideas that speech can't," said Ayako Belka ’25, a biology and psychology major from New Orleans."Gesturing is a form of communication that is so routine to us that we hardly notice it, yet it serves a huge role in daily interactions by communicating emotions and ideas that speech can't," said Ayako Belka ’25, a biology and psychology major from New Orleans. Image Credit: Deb Grove

Ayako Belka ’25

Majors: Biology and Psychology

Hometown: New Orleans

Did anything surprise you about this research? 

What surprised me most about this research was the sheer amount of subconscious gesturing that we do every day. Gesturing is a form of communication that is so routine to us that we hardly notice it, yet it serves a huge role in daily interactions by communicating emotions and ideas that speech can't. Even over a virtual interface such as Zoom, we still saw significant evidence that people were continuing to make meaningful gestures. Over four videos (all around 1.5 hours), we saw 1,497 gestures. That’s 1,497 cases of communication where the person performing it may not even be aware they were doing so. The impact of gestures on the person they are directed to is something we are planning on investigating, but I would also like to find out how gestures impact the person performing them. What does gesturing bring to the person performing the gesture? 

"I am interested in utilizing eye-tracking equipment in visualizing how the attention was changed when partial gestures are presented in comparison with the presentation of full gestures," said Daisy Zang ’23, a psychology major from Kunshan, China."I am interested in utilizing eye-tracking equipment in visualizing how the attention was changed when partial gestures are presented in comparison with the presentation of full gestures," said Daisy Zang ’23, a psychology major from Kunshan, China. Image Credit: Deb Grove

Daisy Zang ’23

Major: Psychology

Hometown: Kunshan, China

What inspired you to take on this research?

During the pandemic, I was struggling with online classes and the time difference, but I was also unsure about why the efficiency of learning was lower in online classes. Therefore, I was very excited to have the opportunity to explore what may impact the class efficiency. In addition, I am also interested in utilizing eye-tracking equipment in visualizing how the attention was changed when partial gestures are presented in comparison with the presentation of full gestures. Our follow-up study plans to start to use eye-tracking equipment to track attention changes of participants when watching partial gestures in comparison with the presentation of full gestures.

"Gestures are unique components of teaching which are usually ignored by the audience, but facilitate our understanding unconsciously," said  Wendy Huang ’25, a sociology and mathematics major from Nanjing, China."Gestures are unique components of teaching which are usually ignored by the audience, but facilitate our understanding unconsciously," said Wendy Huang ’25, a sociology and mathematics major from Nanjing, China. Image Credit: Deb Grove

Wendy Huang ’25

Majors: Sociology and Mathematics

Hometown: Nanjing, China

What inspired you to take on this research?

I am very interested in Zoom fatigue and how this affects the efficiency of online teaching. Gestures are unique components of teaching which are usually ignored by the audience, but facilitate our understanding unconsciously. Hence, I think it is significant to investigate how the Zoom frame’s restriction will affect people’s perception of gestures. 

" Gesturing is a form of communication that is so routine to us that we hardly notice it, yet it serves a huge role in daily interactions by communicating emotions and ideas that speech can't."

Original source can be found here.