The human face is an emotional “hot spot”. New-born babies seem to gravitate spontaneously towards the face of the caretaker. The human face is an emotionally expressive display that is more than the sum of its parts. The face forms a total configuration that manifests a person’s humanity in a way especially engaging to another person. We humans seem to be hard-wired to interact with faces as the location for emotional expression – and the lack of expression. Freud famously said that “betrayal oozes at every pore.” Though Freud’s quip is not about his patient Dora’s facial expression as such, his slogan applies to the face in ways he would have appreciated and which were being explored by Darwin (1871) when Freud was still only fifteen years old. This is where the game gets interesting.

Innovations in computing hardware power, social networking, neural networks, and pattern recognition, are advancing the automated understanding of the expression of human facial emotions. Enter Affectiva (www.Affectiva.com), which, as the saying goes, is disrupting the disrupters. Affectiva was founded in 2009 by Rana el Kaliouby and Rosalind W. Picard, scientists at the MIT Media Lab. In a conversation with Daniel McDuff, Ph.D., principal scientist at Affectiva (www.affectiva.com), I had an opportunity to learn how innovations in the computer-mediated assessment of the emotions are being implemented and brought to market in a variety of applications in advertising and media measurement (Affectiva’s current chosen market), law enforcement, and engaging diseases of empathy. One innovation that Dr McDuff brought to Affectiva from his work at the MIT Media Lab was the use of webcams to collect facial data from persons providing informed consent. This has enabled Affectiva to build a Big Data database of facial expressions to power the processing of its software algorithms. Simply stated, the output of the software process is as assessment of the individual’s emotional experience along a number of dimensions and variables. The devil is in the details. Layers of technology go into Affectiva’s Affdex system to capture and feed information to its algorithms. The system tracks and processes the texture of the human face, performing a complicated mapping to minute facial muscle movements described by the Facial Action Coding System (FACS), resulting in inferences about the categorization and intensity of emotional engagement, valence, and related nuances of affect. As with any software system, issues of ease of use, accessibility and flexibility of the human-machine interface, scalability, maintainability, and end-to-end system integration are front and center. This is where Affectiva seems to have stolen a march on the competition with the use of small computer-based webcams to capture data that is then stored to a Big Data backend. Webcams are pervasive. The potential amount of data is formidable. I have been known to say: “We don’t need more data; we need expanded empathy.” However, sometimes we need both. This seems to be one of those occasions. The deep background to Affectiva’s work is to be found in Paul Ekman (1992) and his colleague Wallace Friesen, who themselves relied on the researches of Charles Darwin (1871) and Silvan S. Tomkins (1961, 1962, 1992, 1993) and Duchenne de Boulogne (1862). In an enormous research effort lasting some eight years, Ekman led a team that coded some 5000 detailed movements of muscles in the face that are activated, in many cases involuntarily, in the arousal of some seven basic emotions. This Facial Action Coding Scheme becomes the basis for software automation. What’s so innovative about that? Well, anyone can try to fake a smile, pretending to be happy when one is really miserable. But what one cannot fake is activation of the “smile muscles” around one’s eyes, which are only engaged by an authentic and positive emotion that expresses one’s sincere delight and which remain uninvolved in an insincere baring of one’s teeth. Unlike one’s lips, which can be voluntarily displayed in a grimace, the muscles around the eyes are not subject to voluntary control. Hence, the opportunity exists for “betrayal to ooze out at every pore”. Moreover, the activation of such a muscle can occur and vanish in a fraction of a second. It moves rapidly across the face at a speed that lies beneath the threshold of one’s ability to see it without significantly slowing down the digital recording and playing it back. Yet the emotion occurs, however briefly. It lives. Such a detailed, minute muscle activation is called a “micro expression”. Micro expressions are hypothesized to be the basis for the enigmatic smile of Leonardo da Vinci’s painting of the Mona Lisa. In our time, Ekman’s facial action coding scheme of micro expressions becomes the basis for detecting deceit in the marketplace, politics and marriage (which, incidentally, is the subtitle of Ekman’s Telling Lies (1992)). In some cases, the marketplace hype is justified, and, in this case, caused me to chase the “rumor of empathy”. Ekman is on record as saying that he is skeptical about empathy, and I do not aim to change that here. In any case, Ekman identifies “duping delight” as the micro expression of happiness of the liar at having “put one over” on the teacher with the deception of being believed that the “dog really did eat the homework”. Or in law enforcement, the micro expression of contempt on the otherwise emotionless face of the would-be terrorist at striking back at the “running dogs of western imperialism”. What would the detection of such micro expressions be if not a subtle example of empathy or, more precisely, empathic receptivity? Before Ekman – and even before Darwin – the philosopher David Hume wrote of a “delicacy of sympathy” (1741), in detecting an impression of which another person was unaware. The word “empathy” had not yet been invented. Close enough. Take aways include: No market, no mission: Affectiva has traction in the advertising and marketing verticals. While all the usual disclaimers apply, and I have not “test driven” the Affdex system, Affectiva seems to be well on its way to integrating its facial recognition algorithm(s) in a comprehensive end-to-end automated process that incudes a user friendly frontend (webcam) and big data backend. The intellectual property is relevant, and patent the algorithms. But absent an integrated, usable approach, it is going to be an idle wheel that does not move any other part of the business process. While a strong start is no guarantee of long term success, Affectiva has innovative technology and a compelling message that resonates with corporate needs to spend money on advertising that delivers demonstrable bang for the buck.
An implied definition of empathy: An account of empathy exists here based on micro expressions, which is what inspired my interest. Though the debate about the relevance of empathy continues, empathy is hypothesized to be at the basis of the human ability (1) to relate emotionally to other persons like oneself (2) to experience other persons as intentional agents (3) to attribute a mind such as one’s own – as in “mindedness” – to other persons like oneself. Empathy is not reducible to emotional contagion, shared-joint attention or mindedness; but at least the first two are input for further empathic understanding, empathic interpretation, and empathic responsiveness that enriches a person’s relations with other human beings. Absent empathy, people cease to matter to the person lacking empathy, though people may be useful in certain means-ends way of providing services. In disorders of empathy, one or more of these mechanisms has misfired or is hypothesized to be missing. The individual “on the autism spectrum” seems to be unaware of the emotions, intentions or mindedness of other people. The subsequent breakdowns in human development, education, and day-to-day functioning are debilitating in the extreme and can even be life threatening. People and computer systems can produce similar results and output using profoundly different means and methods. Though it is improbable that the Affdex software arrives at its conclusions about what people are experiencing emotionally in the same way that the human brain arrives at its results, the possibilities for comparison are significant. A comparison between the steps of human brain-based empathy and the artificial empath implemented in software may reveal what can go wrong and suggest meaningful interventions. Finally, no substitute exists for an expert clinical differential diagnosis by an informed human – though hope springs eternal in the matter of eliminating focus groups in marketing – but the use of software to detect and analyze micro expressions – or their absence – can be a significant check and balance in the diagnostic process. Empathy is still on the short list of those things where humans enjoy a decisive advantage over automated systems. Still, it is sobering the way the boundary keeps getting pushed around as humans are beaten by computing systems in chess, natural language processing in Jeopardy, and now challenged by decoding facial emotions. In any case, the combination of human judgment and a software system acting as a kind of “co-pilot” to the decision-making human is a compelling partnership. In summary, the rumor of empathy at Affectiva is confirmed. Empathy lives at Affectiva. I hasten to add that “a rumor of empathy” is my turn of phrase – my spin – and not a description employed by Affectiva, though I suggest that the distinction is an apt one.

Relevant URLs