Lip Reading Computer Interprets Human Emotions

Author: Ian C. Langtree - Writer/Editor for Disabled World (DW)
Published: 10 Sep 2012 - Updated: 30 Jun 2026
Publication Type: Informative

Table of Contents:
Synopsis - Definition - Introduction - Main - Insights, Updates - Related Publications

Synopsis: This research, published in the International Journal of Artificial Intelligence and Soft Computing, describes a computer system that interprets human emotions by analyzing the shape and pattern of a person's lips, using a genetic algorithm that improves with each iteration to fit irregular ellipse equations to the mouth. Developed by Karthigayan Muthukaruppan of Manipal International University in Malaysia and colleagues, the system was trained on photographs of people from South-East Asia and Japan and can classify six widely accepted emotions - happiness, sadness, fear, anger, disgust, and surprise - along with a neutral expression. The work is useful because it points toward practical assistive applications, particularly helping people who lack speech to interact more effectively with computer-based communication devices such as voice synthesizers.

At a Glance

• 1 - The upper and lower lip are each analyzed by the algorithm as two separate ellipses.
• 2 - The genetic algorithm refines its accuracy with each iteration as it matches equations to mouth shapes.
• 3 - Recognizing emotion from a real human face remains difficult, since deeper feelings also show through the eyes and eyebrows rather than the mouth alone.

Topic Definition: Emotion Recognition

Emotion recognition is the process by which a computer or other system identifies a person's emotional state by analyzing observable signals such as facial expressions, lip and mouth shape, eye and eyebrow movement, voice, or other physical cues. In the context of facial analysis, the system measures features of the face and matches them against patterns associated with established emotions such as happiness, sadness, fear, anger, disgust, and surprise, along with a neutral state. As an assistive technology, emotion recognition can help people who lack speech communicate more naturally, allowing communication devices to respond to a user's feelings rather than words alone.

Introduction

Computer, read my lips - Emotion detector developed using a genetic algorithm. A computer is being taught to interpret human emotions based on lip pattern, according to research published in the International Journal of Artificial Intelligence and Soft Computing.

The system could improve the way we interact with computers and perhaps allow disabled people to use computer-based communications devices, such as voice synthesizers, more effectively and more efficiently.

Main Content

Karthigayan Muthukaruppanof Manipal International University in Selangor, Malaysia, and co-workers have developed a system using a genetic algorithm that gets better and better with each iteration to match irregular ellipse fitting equations to the shape of the human mouth displaying different emotions.

They have used photos of individuals from South-East Asia and Japan to train a computer to recognize the six commonly accepted human emotions - happiness, sadness, fear, angry, disgust, surprise - and a neutral expression. The upper and lower lip is each analyzed as two separate ellipses by the algorithm.

"In recent years, there has been a growing interest in improving all aspects of interaction between humans and computers especially in the area of human emotion recognition by observing facial expression," the team explains.

Earlier researchers have developed an understanding that allows emotion to be recreated by manipulating a representation of the human face on a computer screen. Such research is currently informing the development of more realistic animated actors and even the behavior of robots. However, the inverse process in which a computer recognizes the emotion behind a real human face is still a difficult problem to tackle.

It is well known that many deeper emotions are betrayed by more than movements of the mouth. A genuine smile for instance involves flexing of muscles around the eyes and eyebrow movements are almost universally essential to the subconscious interpretation of a person's feelings. However, the lips remain a crucial part of the outward expression of emotion. The team's algorithm can successfully classify the seven emotions and a neutral expression described.

The researchers suggest that initial applications of such an emotion detector might be helping disabled patients lacking speech to interact more effectively with computer-based communication devices, for instance.

"Lip pattern in the interpretation of human emotions " in Int. J. Artificial Intelligence and Soft Computing, 2012, 3, 95-107

How Lip-Reading Errors Happen, Revealed by Network Science: University of Kansas researchers mapped 20,000 English words to show why some are far harder to read on the lips than others.

Insights, Analysis, and Developments

Author Credentials: Ian is the founder and Editor-in-Chief of Disabled World, a leading resource for news and information on disability issues. With a global perspective shaped by years of travel and lived experience, Ian is a committed proponent of the Social Model of Disability-a transformative framework developed by disabled activists in the 1970s that emphasizes dismantling societal barriers rather than focusing solely on individual impairments. His work reflects a deep commitment to disability rights, accessibility, and social inclusion. To learn more about Ian's background, expertise, and accomplishments, visit his full biography.