TORONTO, March 30, 2023 – Visual cues people normally pick up when communicating in-person can become misleading and false over video platforms like Zoom and Skype, making communication not only more difficult, but also exhausting, says new research out of York University.

Vision Professor Nikolaus Troje of York’s Faculty of Science points out that a wealth of nuanced non-verbal communication cues that enrich verbal communication in the real world, are misleading and disruptive during video communication. This is part of why people experience Zoom fatigue.

“The problem is that directional information, eye gaze in particularly, gets disrupted in video calls,” says Troje, Canada Research Chair in Reality Research and director of the BioMotion Lab at York’s Centre for Vision Research. “In real life, if I meet someone’s eyes, I can rely on the fact that the other person experiences the same. The other person knows that I look at her and I know that she looks at me.”

Social gaze informs bonding, trust and rapport, moderates how people communicate emotions through facial expressions and negotiates turn taking. All of this affects the efficiency of communication.

“Social gaze during communication is a sophisticated dance during which eye contact is established and broken repeatedly,” says Troje. “Eye contact events trigger physiological responses, such as changes in pupil size, heart rate and skin conductance, in both partners and synchronize their affective states.”

If these visual cues are absent, such as in a phone conversation, people are able to adjust to their absence. The problem with video communication is that they are present, but misleading and deceptive. “The visual system and the rest of our body respond to them even though they are no longer faithful. Even though I may experience eye contact, my partner may not even see me, as she is looking into her camera instead.

“Real face-to-face meetings are way more efficient, productive and less exhausting,” says Troje. “This lack of efficiency using internet-based video communication extends to teaching, learning, employment interviews and therapy.”

Solving the problem

Manipulating only the eyes in the video feed, as some new video filters do, does not fix the main problem and cannot re-establish natural, dynamic eye contact behaviour.

In analysing the problem, Troje also points to solutions. New generations of video conferencing platforms need to be able to track user’s head location in real time, without lags, to drive a virtual camera. AI and machine learning can then be used to generate the changing views of the communication partner.

“Given the advent of fast on-board computer vision chips and new algorithms in computer vision and machine learning, that approach is feasible" says Troje. "The societal implications of fixing the issues are huge. Trust and rapport building through functional eye contact is essential for telehealth and business meetings alike.”

The paper, Zoom disrupts eye contact behaviour: problems and solutions, was published today in the journal Trends in Cognitive Sciences.