Month: March 2022

The Application of Facial Expression and Micro-expression in Virtual Character Emotional Expressions

Draft of Literature Review

Introduction:

Emotion can be an important part of performance animation because it can build up the relationships between characters and the audience. Traditionally, performance animations are based on references and regular patterns, while ignoring the behavioral and psychological exploration of the relationship among emotions and facial expressions. Such method may weaken the expression of emotional details. Based on subjects such as psychology and evolutionary-biology, scientists believe that facial expressions and micro-expressions have a certain connection to human emotions, and this connection can be decoded in a scientific way. Their research provides basis for virtual characters realistic emotional expressions. This paper will base on recent research on facial expressions, micro-expressions and animation techniques, analyzing real cases in film, animation and practices, and try to find an artistic-scientific blended way to build character emotions in performance animations.

Literature review:

Key words: Facial expressions, Emotions, Animation.

Traditional performance animation:

According to Norman McLaren, “Animation is not the art of drawings that move but the art of movements that are drawn”. 2d and 3d animation can affect humans and their emotions significantly. Traditionally, animators animate performance animation based on references or their experience of observing human behaviors. Jason Osipa shares a large number of computer facial animation techniques in his book Stop Starring. The book covers variable kinds of facial expression techniques that animators used in their productions, which also covers techniques on emotion expressions (Osipa, 2010). However, these principles mostly base on observations but ignored the research on scientific level. Moreover, in certain emotional situations like hiding true emotions or deception, it is difficult to find patterns of people’s emotional changes or even represent them through traditional methods. However, some scientific or psychological research can provide materials of these specific emotional expressions for animators. Therefore, human psychology and other science research can be a guidance to help animators to develop correct animations for their audiences to connect with.

The scientific analysis development of facial expression and micro-expressions:

The research on the relationship among emotions, facial expressions and micro expressions has lasted for decades. Several scholars’ developments have been considered as the foundation of this area, and part of them has already been applied into film and animation industry.

Facial expression:

The research on facial expressions started by Charles Darwin and then refined by other scholars. Neuropsychological studies point out the asymmetry of facial expressions, which means that the two sides of human face are not performed symmetrically when emotional expressions occurs. Additionally, scientists realized that emotions can be recognized more easily on the left part of human face. They mentions that socially appropriate signals are clearly visible on the right face, while personalized signals are visible on the left face (Mandal and Awasthi, 2015, p. 274). Other scientists, for example, Ekman and Friesen, developed the universality thesis of facial expressions, which refers to accurate recognition of facial expressions across cultures at better-than-chance levels (Ekman et al. 1987). Ekman also proposed the idea of six basic emotional expressions, which has been accepted by psychologists: happiness, sadness, anger, fear, surprise and disgust (Russell and José Miguel Fernández-Dols, 2002, p. 11). However, it is still being criticized by some cross-cultural studies on emotional facial expressions (Russell, 1994).

Some of scientists then developed several automated system for the recognition of facial expressions. For example, electromyography and electroencephalogram (Mandal and Awasthi, 2015, p. 9). These systems fit the requirement of collecting experiment data and scientific research. However, they are not good guidance for animators. With the development of anatomically based coding systems, however, animators start to have more resources for analyzing facial expressions. Hjortsjö’s Mimic language can be considered as one of the earliest explorations of facial muscular activities systematization. The mimicry covers the additional expressive movements of gestures and postures, which are the characteristic manifestations of emotional states (Hjortsjö, 1970). Hjortsjö also described the facial expressions which are related to twenty four emotions, and divided these expressions into eight categories (Hjortsjö, 1970).

Another important development of anatomically based coding systems is Facial Action Coding System (FACS). FACS is developed by Paul Ekman and Wallace Friesen in 1978. They breaks down facial actions into small parts, which is called action units (AUs). Each of them can be considered as the basic elements of facial expressions. With the combination of different AUs, people can make different kinds of facial expressions based on muscle movements. At first facial action coding system is designed for motion records (Ekman, Friesen and Hager, 2002). However, it has now been widely applied into film industry and computer animation for decades (Parke and Waters, 2020, p. 33).

Micro-expressions:

Micro-expression is considered as a kind of more typical facial expression of emotion. The definition of micro-expression is still not clear. Mark G. Frank and Elena Svetieva tend to define micro-expression as any expression of emotion that is shown at 0.5 s or less because previous research had suggested that most of spontaneous expressions of emotion lasts between 0.5 and 4 (or 5) seconds (Mandal and Awasthi, 2015, p. 229). A distinctive feature of micro-expressions is that they reveal true emotional states in the form of facial expression for a very short period of time (0.5s or less), which is then quickly disguised or suppressed by another one.

The relationship between micro-expression and deception has been analyzed by some scholars. Some of the research focusing on the difference between real smile and fake smile. Scientists found that the main difference between real smile and fake smile is the muscle movements around eye area (Duchenne, 1990). Although muscles on the mouth corners are pulled up in both of these kinds of smiles, only the real smile will trigger the movements of orbicularis oculi muscle around the eyes (Duchenne, 1990). Additionally, according to DePaulo’s research, the facial pleasantness of liars is much lower than normal level. Liars will have more chin raises, more lip pressing, and look more nervous (DePaulo et al. 2003). However, other facial expressions like smiling, eyebrow lowering or raising have not shown consistently significant effect sizes (DePaulo et al. 2003).

References:

Carl-Herman Hjortsjö (1970). Man’s face and mimic language. Studentlitteratur.

DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003). “Cues to deception”, Psychological Bulletin, pp. 74–118.

Duchenne, G. B. (1990). The Mechanism of Human Facial Expression. Edited by R. A. Cuthbertson. Cambridge: Cambridge University Press (Studies in Emotion and Social Interaction). doi: 10.1017/CBO9780511752841.

Ekman, P., Friesen, W.V. and Hager, J.C. (2002). Facial action coding system. Salt Lake City: Research Nexus.

Ekman, P., Friesen, W. V., O’Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., et al. (1987). “Universals and cultural differences in the judgments of facial expressions of emotion”, Journal of Personality and Social Psychology, 53(4), pp. 712–717.

Mandal, M.K. and Awasthi, A. (2015). Understanding Facial Expressions in Communication: Cross-cultural and Multidisciplinary Perspectives. New Delhi: Springer India.

Osipa, J. (2010). Stop staring facial modeling and animation done right. Indianapolis, Ind Sybex.

Parke, F.I. and Waters, K. (2020). Computer facial animation. Boca Raton: Crc Press.

Russell, J. A. (1994). “Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies”, Psychological Bulletin, pp. 102–141.‌

Russell, J.A. and José Miguel Fernández-Dols (2002). The psychology of facial expression. Cambridge: Cambridge University Press.

The application of body language and micro-expressions in virtual character emotional expressions

1.Introduction: Emotion can be an important part of performance animation because it can build up the relationships between characters and the audience. Based on subjects such as psychology and evolutionary-biology, scientists believe that body language and micro-expressions have a certain connection to human emotions, and this connection can be decoded in a scientific way. Their research provides the basis for developing the computational modeling of face from automated recognition to make expressions of emotions possible among virtual characters. This paper will base on recent research on body language, micro-expressions and animation techniques, analyzing real cases in film, animation and practices, and trying to find an artistic-scientific blended way to build character emotions in performance animations.

Key words: Facial expressions, Emotions, Animation.

2.Methodology:

3.Main body:

The scientific and psychological analysis development of facial expression and body language:

Guillaume Duchenne’s development:

The most remarkable investigation of facial expression of its time was by Guillaume Duchenne. It is remarkable because he documented his scientific research with the then-new medium of photography in the 1860s. He investigated facial articulation by stimulating facial muscles with moist electrodes that delivered direct “galvanic” current to key motor points on the surface of face. More recently, Duchenne’s approach of transcutaneous electrical nerve stimulation has been adopted in performance art where a performer’s face can be controlled via computer programs such as Text-to-Speech, with some intriguing results.

The Mimic language:

The Mimic language developed by Hjortsjo is one of the earliest attempts to investigate and systematize the muscular activities that create the diverse facial expressions. Hjortsjo’s motivation was to develop a language for describing facial expression. According to Hjortsjo, mimicry includes the play of facial features, gestures, and postures.

The concept of mimicry includes additional expressive movements in the form of gestures and postures that are characteristic manifestations of emotional states. Hjortsjo refers to these movements as the mimic co-movements, which include movements of the jaw, the neck, the shoulders, the arms, and the hands.

The words of the Mimic language correspond to facial expressions. These words, or expressions, are formed by combining the letters of the language the actions of the mimic muscles and the mimic co-movements. Hjortsjo describes the facial expressions corresponding to twenty-four emotions. These expressions are arranged in eight groups.

Facial Action Coding System (FACS):

The Facial Action Coding System (FACS), developed by Paul Ekman and Wallace Friesen in 1978, breaks down facial actions into small units called action units (AUs). Each AU represents an individual muscle action, or an action of a small group of muscles, a single recognizable facial posture. In total, FACS classifies 66 AUs that in combination could generate defined and gradable facial expressions. As a result, FACS has been used extensively in facial animation over the past decade to help animators interpret and construct realistic facial expressions. The FACS describes the set of all possible basic AUs performable by the human face. According to Ekman, “FACS allows the description of all facial behavior we have observed, and every facial action we have attempted.”

The analysis of body language and micro-expressions in real practices:

I may analyze some examples in feature films or game area to figure out how these characters use their body language and micro-expressions to represent their emotions in layers. The exmples may comes from some works’ sequences which contains various emotions or even multi-layer emotions. For example, The Last of US series, Detroit: BecomeHuman. Se7en, The Silence of the Lambs. I may also use some bad examples and analyse their problems to draw a critical conclusion.

The exploration of virtual character’s body language and micro-expressions:

I may introduce some testing or experimental performance animation by myself, analyse them and point out the problems.

Experimental performance animation refinement and summary:

Conclusions:

blabla

References:

Mandal, M.K., Awasthi, A. (2015). Understanding Facial Expressions in Communication : Cross-cultural and Multidisciplinary Perspectives. New Delhi: Springer India.

Ekman, P., Friesen, W.V. and Hager, J.C. (2002). Facial action coding system. Salt Lake City: Research Nexus.

Osipa, J. (2010). Stop staring facial modeling and animation done right. Indianapolis, Ind Sybex.

Martinez, L., Falvello, V.B., Aviezer, H. and Todorov, A. (2015). “Contributions of facial expressions and body language to the rapid perception of dynamic emotions.”, Cognition and Emotion, 30(5), pp. 939–952.

Kovecses, Z. (2000). Metaphor and emotion : language, culture, and body in human feeling. Cambridge: Cambridge University Press.

Russell, J.A. and José Miguel Fernández-Dols (2002). The psychology of facial expression. Cambridge: Cambridge University Press.