Category: Final Major Project & Thesis

The Application of Facial Expression and Micro-expression in Virtual Character Emotional Expressions

Draft of Literature Review

Introduction:

Emotion can be an important part of performance animation because it can build up the relationships between characters and the audience. Traditionally, performance animations are based on references and regular patterns, while ignoring the behavioral and psychological exploration of the relationship among emotions and facial expressions. Such method may weaken the expression of emotional details. Based on subjects such as psychology and evolutionary-biology, scientists believe that facial expressions and micro-expressions have a certain connection to human emotions, and this connection can be decoded in a scientific way. Their research provides basis for virtual characters realistic emotional expressions. This paper will base on recent research on facial expressions, micro-expressions and animation techniques, analyzing real cases in film, animation and practices, and try to find an artistic-scientific blended way to build character emotions in performance animations.

Literature review:

Key words: Facial expressions, Emotions, Animation.

Traditional performance animation:

According to Norman McLaren, “Animation is not the art of drawings that move but the art of movements that are drawn”. 2d and 3d animation can affect humans and their emotions significantly. Traditionally, animators animate performance animation based on references or their experience of observing human behaviors. Jason Osipa shares a large number of computer facial animation techniques in his book Stop Starring. The book covers variable kinds of facial expression techniques that animators used in their productions, which also covers techniques on emotion expressions (Osipa, 2010). However, these principles mostly base on observations but ignored the research on scientific level. Moreover, in certain emotional situations like hiding true emotions or deception, it is difficult to find patterns of people’s emotional changes or even represent them through traditional methods. However, some scientific or psychological research can provide materials of these specific emotional expressions for animators. Therefore, human psychology and other science research can be a guidance to help animators to develop correct animations for their audiences to connect with.

The scientific analysis development of facial expression and micro-expressions:

The research on the relationship among emotions, facial expressions and micro expressions has lasted for decades. Several scholars’ developments have been considered as the foundation of this area, and part of them has already been applied into film and animation industry.

Facial expression:

The research on facial expressions started by Charles Darwin and then refined by other scholars. Neuropsychological studies point out the asymmetry of facial expressions, which means that the two sides of human face are not performed symmetrically when emotional expressions occurs. Additionally, scientists realized that emotions can be recognized more easily on the left part of human face. They mentions that socially appropriate signals are clearly visible on the right face, while personalized signals are visible on the left face (Mandal and Awasthi, 2015, p. 274). Other scientists, for example, Ekman and Friesen, developed the universality thesis of facial expressions, which refers to accurate recognition of facial expressions across cultures at better-than-chance levels (Ekman et al. 1987). Ekman also proposed the idea of six basic emotional expressions, which has been accepted by psychologists: happiness, sadness, anger, fear, surprise and disgust (Russell and José Miguel Fernández-Dols, 2002, p. 11). However, it is still being criticized by some cross-cultural studies on emotional facial expressions (Russell, 1994).

Some of scientists then developed several automated system for the recognition of facial expressions. For example, electromyography and electroencephalogram (Mandal and Awasthi, 2015, p. 9). These systems fit the requirement of collecting experiment data and scientific research. However, they are not good guidance for animators. With the development of anatomically based coding systems, however, animators start to have more resources for analyzing facial expressions. Hjortsjö’s Mimic language can be considered as one of the earliest explorations of facial muscular activities systematization. The mimicry covers the additional expressive movements of gestures and postures, which are the characteristic manifestations of emotional states (Hjortsjö, 1970). Hjortsjö also described the facial expressions which are related to twenty four emotions, and divided these expressions into eight categories (Hjortsjö, 1970).

Another important development of anatomically based coding systems is Facial Action Coding System (FACS). FACS is developed by Paul Ekman and Wallace Friesen in 1978. They breaks down facial actions into small parts, which is called action units (AUs). Each of them can be considered as the basic elements of facial expressions. With the combination of different AUs, people can make different kinds of facial expressions based on muscle movements. At first facial action coding system is designed for motion records (Ekman, Friesen and Hager, 2002). However, it has now been widely applied into film industry and computer animation for decades (Parke and Waters, 2020, p. 33).

Micro-expressions:

Micro-expression is considered as a kind of more typical facial expression of emotion. The definition of micro-expression is still not clear. Mark G. Frank and Elena Svetieva tend to define micro-expression as any expression of emotion that is shown at 0.5 s or less because previous research had suggested that most of spontaneous expressions of emotion lasts between 0.5 and 4 (or 5) seconds (Mandal and Awasthi, 2015, p. 229). A distinctive feature of micro-expressions is that they reveal true emotional states in the form of facial expression for a very short period of time (0.5s or less), which is then quickly disguised or suppressed by another one.

The relationship between micro-expression and deception has been analyzed by some scholars. Some of the research focusing on the difference between real smile and fake smile. Scientists found that the main difference between real smile and fake smile is the muscle movements around eye area (Duchenne, 1990). Although muscles on the mouth corners are pulled up in both of these kinds of smiles, only the real smile will trigger the movements of orbicularis oculi muscle around the eyes (Duchenne, 1990). Additionally, according to DePaulo’s research, the facial pleasantness of liars is much lower than normal level. Liars will have more chin raises, more lip pressing, and look more nervous (DePaulo et al. 2003). However, other facial expressions like smiling, eyebrow lowering or raising have not shown consistently significant effect sizes (DePaulo et al. 2003).

References:

Carl-Herman Hjortsjö (1970). Man’s face and mimic language. Studentlitteratur.

DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003). “Cues to deception”, Psychological Bulletin, pp. 74–118.

Duchenne, G. B. (1990). The Mechanism of Human Facial Expression. Edited by R. A. Cuthbertson. Cambridge: Cambridge University Press (Studies in Emotion and Social Interaction). doi: 10.1017/CBO9780511752841.

Ekman, P., Friesen, W.V. and Hager, J.C. (2002). Facial action coding system. Salt Lake City: Research Nexus.

Ekman, P., Friesen, W. V., O’Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., et al. (1987). “Universals and cultural differences in the judgments of facial expressions of emotion”, Journal of Personality and Social Psychology, 53(4), pp. 712–717.

Mandal, M.K. and Awasthi, A. (2015). Understanding Facial Expressions in Communication: Cross-cultural and Multidisciplinary Perspectives. New Delhi: Springer India.

Osipa, J. (2010). Stop staring facial modeling and animation done right. Indianapolis, Ind Sybex.

Parke, F.I. and Waters, K. (2020). Computer facial animation. Boca Raton: Crc Press.

Russell, J. A. (1994). “Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies”, Psychological Bulletin, pp. 102–141.‌

Russell, J.A. and José Miguel Fernández-Dols (2002). The psychology of facial expression. Cambridge: Cambridge University Press.

The application of body language and micro-expressions in virtual character emotional expressions

1.Introduction: Emotion can be an important part of performance animation because it can build up the relationships between characters and the audience. Based on subjects such as psychology and evolutionary-biology, scientists believe that body language and micro-expressions have a certain connection to human emotions, and this connection can be decoded in a scientific way. Their research provides the basis for developing the computational modeling of face from automated recognition to make expressions of emotions possible among virtual characters. This paper will base on recent research on body language, micro-expressions and animation techniques, analyzing real cases in film, animation and practices, and trying to find an artistic-scientific blended way to build character emotions in performance animations.

Key words: Facial expressions, Emotions, Animation.

2.Methodology:

3.Main body:

The scientific and psychological analysis development of facial expression and body language:

Guillaume Duchenne’s development:

The most remarkable investigation of facial expression of its time was by Guillaume Duchenne. It is remarkable because he documented his scientific research with the then-new medium of photography in the 1860s. He investigated facial articulation by stimulating facial muscles with moist electrodes that delivered direct “galvanic” current to key motor points on the surface of face. More recently, Duchenne’s approach of transcutaneous electrical nerve stimulation has been adopted in performance art where a performer’s face can be controlled via computer programs such as Text-to-Speech, with some intriguing results.

The Mimic language:

The Mimic language developed by Hjortsjo is one of the earliest attempts to investigate and systematize the muscular activities that create the diverse facial expressions. Hjortsjo’s motivation was to develop a language for describing facial expression. According to Hjortsjo, mimicry includes the play of facial features, gestures, and postures.

The concept of mimicry includes additional expressive movements in the form of gestures and postures that are characteristic manifestations of emotional states. Hjortsjo refers to these movements as the mimic co-movements, which include movements of the jaw, the neck, the shoulders, the arms, and the hands.

The words of the Mimic language correspond to facial expressions. These words, or expressions, are formed by combining the letters of the language the actions of the mimic muscles and the mimic co-movements. Hjortsjo describes the facial expressions corresponding to twenty-four emotions. These expressions are arranged in eight groups.

Facial Action Coding System (FACS):

The Facial Action Coding System (FACS), developed by Paul Ekman and Wallace Friesen in 1978, breaks down facial actions into small units called action units (AUs). Each AU represents an individual muscle action, or an action of a small group of muscles, a single recognizable facial posture. In total, FACS classifies 66 AUs that in combination could generate defined and gradable facial expressions. As a result, FACS has been used extensively in facial animation over the past decade to help animators interpret and construct realistic facial expressions. The FACS describes the set of all possible basic AUs performable by the human face. According to Ekman, “FACS allows the description of all facial behavior we have observed, and every facial action we have attempted.”

The analysis of body language and micro-expressions in real practices:

I may analyze some examples in feature films or game area to figure out how these characters use their body language and micro-expressions to represent their emotions in layers. The exmples may comes from some works’ sequences which contains various emotions or even multi-layer emotions. For example, The Last of US series, Detroit: BecomeHuman. Se7en, The Silence of the Lambs. I may also use some bad examples and analyse their problems to draw a critical conclusion.

The exploration of virtual character’s body language and micro-expressions:

I may introduce some testing or experimental performance animation by myself, analyse them and point out the problems.

Experimental performance animation refinement and summary:

Conclusions:

blabla

References:

Mandal, M.K., Awasthi, A. (2015). Understanding Facial Expressions in Communication : Cross-cultural and Multidisciplinary Perspectives. New Delhi: Springer India.

Ekman, P., Friesen, W.V. and Hager, J.C. (2002). Facial action coding system. Salt Lake City: Research Nexus.

Osipa, J. (2010). Stop staring facial modeling and animation done right. Indianapolis, Ind Sybex.

Martinez, L., Falvello, V.B., Aviezer, H. and Todorov, A. (2015). “Contributions of facial expressions and body language to the rapid perception of dynamic emotions.”, Cognition and Emotion, 30(5), pp. 939–952.

Kovecses, Z. (2000). Metaphor and emotion : language, culture, and body in human feeling. Cambridge: Cambridge University Press.

Russell, J.A. and José Miguel Fernández-Dols (2002). The psychology of facial expression. Cambridge: Cambridge University Press.

This week we learned the basic of experimental animations and watched several videos to strengthen the idea of this type of animation. One of these animations impressed me a lot which is An Optical Poem, made by Oskar Fischinger in 1938.

Frankly, I am not some one who can really appreciate experimental animations since I never done this before, but I am deeply impressed by Oskar Fischinger’s work after I saw it. It is hard to imagine how the artist made this masterpiece 80 years ago. The changing of these shapes perfectly fits the rhythm and the color design of the whole scene is powerful and impressive. One interesting is that recently this kind of animation form is widely used in some music games, which can be considered as an extension of this kind of experimental style. The idea of this animation is ahead of the times, and it still looks amazing after years.

I think the reason why it can be considered as an experimental animation is that it talks about the relationship among shapes, colors and sounds. It is a great try to mix these things together to bring the audience a great experience on watching or listening. This work can also be considered as a remaster of the music, which contains the ideas and attitude of Oskar towards this music. Although the form of this demonstration can be little abstract, we can still understand the mood inside of this work.

Consider the following questions and try to provide brief answers on your blog for next week.

• On graduation which area or environment of production do you wish to focus upon and why?
• What skills will you need to attain the standards required for vocational practice?
• How will you showcase your FMP practice for the final shows?
• Is it important to directly connect the thesis research to your practical work?
• Do you have an area of research you wish to conduct that is unrelated to practical element?

There are several areas that I would like to focus on in my thesis and project:

Area 1: Facial performance animation

Facial performance animation is an area that I would like to develop an in-depth study and research in. I was moved by the facial performance in some CGI films and games like The Last of Us, L.A Noire and Resident Evil series. It has been applied in animation film production for years, and its applications in computer game area is increasing. With the technology updating, it can be forseen that facial performance animation will be used in a wider area in the future.

From my point of view, great facial performance animation can enhance the development of characters, which also builds up a close relationship between viewers/players and characters. The small details in facial animation can covers tones of information, which is more artistic and playful. Although some of realistic facial animations are made by motion capture, I still believe that hand-made facial performance is more impressive and touching.

To enhance facial performance animation, several things must be learned: face anatomy, facial expressions, rigging. Also, learning the basic use of motion capture and thesis of acting are important.

Area 2: Realistic shading for characters

Another topic that I’m interested in is realistic skin shading, which can be also called the subsurface scattering (SSS) skin shading. Realistic skin texture may covers several areas like facial and body details sculpting, muti-channel texture manufacture and rendering.

The research of this area can also be expanded. For example, blended dynamic normal maps, which can be driven by rigged controlers to get a better facial performance result. Another example is the research on the diffrence of realistic shadings between races, which may covers more detailed thesis. What’s more, the relationship between emotions, actions and skin textures (people’s actions and emotions will change the performance of their skin textures. For exmple, when people angry their face may turn red)

To enhance skin shading, several things must be learned: anatomy, SSS shading, textures, skin lighting and rendering. Simple rigging techniques are also required.

Area 3: Facial rigging or other advanced rigging

A better research or practice of character rigging or animal rigging may improve my understanding of softwares, animations and film/game production. Therefore, facial or other advanced rigging techniques can be another area of my thesis writing. These techniques may cover: FACES (Facial Action Coding System) rigging techniques, quadruped creature rigging or other things.