Think back to the last face-to-face conversation you had.
How many of your thoughts and feelings were communicated with words alone? Did you rely on facial expressions or body language to help get your point across?
Most people don’t realize how much non-verbal communication impacts a conversation, but at Emerson’s Facial Affective and Communicative Expressions (FACE) Lab, researchers explore it every day. An initiative run by Dr. Ruth Grossman, a professor in Emerson’s department of Communication Sciences and Disorders, the FACELab is dedicated to studying the mechanisms behind human facial expressions, how facial expressions align with verbal expression, and how both types of communication are perceived in conversation.
Now, thanks to a new grant from the National Institute of Health (NIH), FACELab researchers and their colleagues will spend the next five years using a mix of manual coding and machine-learning technology to evaluate the impact of nonverbal communication in conversations between autistic and neurotypical people.
The FACELab recently received a five-year, $4.5 million grant from the NIH to collaborate on a new research project with the Emotion and Neurodiversity Lab at the Children’s Hospital of Philadelphia (CHOP). They will work together to analyze conversations between autistic and neurotypical people, specifically how their verbal and non-verbal expressions align while conversing, and the subsequent success of those interactions.
The research will build on another NIH-funded, ongoing collaborative study between the FACELab, CHOP, the University of Connecticut, and the University of Aarhus in Denmark. That study involves recording more than 500 adolescent participants, both autistic and neurotypical, having conversations over Zoom and in person. The goal of that study is to determine what verbal factors predict conversational success between those populations.
The new study will use the video recordings of those same conversations to analyze primarily non-verbal communication. Communication Sciences and Disorders Professor and FACE Lab Director Ruth Grossman is a lead researcher on both studies, and said John Herrington of CHOP, her fellow Multiple Principal Investigator on the new grant, first had the idea to expand the research with a new project.
The researchers will study different aspects of multimodal communication, and hope to learn about how people communicate non-verbally with different conversation partners.
“We can look at facial expressions, tone of voice, body posture, gestures, all those things, in relation to the verbal content…and see how well those things align, that’s what my interest is,” she said, “John Herrington is really interested in how the speakers align their nonverbal expressions to each other.”
Grossman, Harrington, and their respective teams of postdoctoral researchers, newly funded by the grant, will use different methods to analyze separate aspects of the videos.
Herrington and his team at CHOP will use a machine learning algorithm they developed to identify micro non-verbal behavior in the videos that is difficult for humans to detect. That behavior might include slight increases in vocal speed, vocal pitch levels, and small facial expressions conversation partners made to accommodate one another.
Meanwhile, Grossman and her fellow Emerson researchers will use manual coding to identify and mark if the words participants are saying in the videos appropriately match their facial expressions. For instance, whether they are smiling while talking about a sad topic. Throughout, the research teams will collaborate on their findings and use both datasets to construct multi-modal profiles of the speakers to determine how those profiles relate to the success of an interaction.
Putting Research Into Action
Grossman said when she first began researching this topic, she hoped to help autistic people better align their voices and facial expressions when speaking to improve their communication skills. Much of her research focuses on social first impressions and the judgements people make about others from split second interactions. In her studies, she said, autistic people are consistently rated less favorably than neurotypical people based on split-second video clips, including by other autistic people.
“I think there is an actual quantifiable difference in what autistic people do with their faces and their voices, and how that’s perceived by other people,” she said, “but we don’t yet know enough about what exactly they’re doing differently, and how that’s driving perceptions.”
Now, she has a new goal with her research. She hopes it can help the world better understand how autistic and nonautistic interactions differ from one another, and the sources of miscommunication between neurotypical and autistic people.
More widespread knowledge about how people of different neurotypes communicate, she said, can help both autistic and non-autistic people communicate better with each other at work, school, and in personal relationships.
“We need to understand how we should and should not read some facial expressions and tone of voice differences, so that we’re not making assumptions about what people’s underlying mental state is, or what their intentions are based on a misread of a facial expression,” she said. “It’s important for autistic people to understand how they’re coming across differently than maybe they want to, and for non-autistic people to understand the same thing.”