Guiltier pleasures: social cognition in gaming
The following is a copy of a post I made to the defunct and currently-unavailable Ludonauts group blog. The original URL was http:/www.ludonauts.com/archives/000038.shtml; I’m also copying the comments. It goes without saying, I would probably write this differently today (mostly by being a little less confident about the narrative of social cognitive development – there are other theories, and the one that I deployed here shouldn’t be read as having universal acceptance.)
The original date of this post was March 30, 2004
I recently had the opportunity to help the creators of Facade demo their game/drama1 at the 2004 GDC. What I saw, when watching people play it, brought into focus for me just why I have been pessimistic about games as a compelling media for work that is as rich in meaning and humanistic depth as film and literature can be; and I think I understand what will have to happen before what we now call “videogames” evolves into something that will become the definitive expressive practice for the next century. It does – and does not – involve the old rhubarbs “narrative” and “character” and “story.” Instead, it’s about cognition – and a shift in just what cognitive fluencies are engaged by the game to navigate game-space.
In Facade, the player has been invited to the upscale apartment of a married couple, Trip and Grace – old college friends whom the player had introduced to each other 10 years ago – for drinks. It quickly becomes apparent that their marriage is in crisis, and the interaction of the player with the characters moves the plot along in a way entirely unlike the typical “choose your own adventure” way. The engine(s) behind the game are interesting enough – there is a semantic parser which passes the player’s statements and actions to the AI’s, which in turn interact with a drama manager that moves the situation and conversations to distinct dramatic “beats.” The interchange between these motivated AI agents and the structures of the drama manager is a compelling resolution of the tension between character-driven and narrative-driven interactive storytelling, but what actually caught my eye was watching players – gamers, most of them, with the expectations and gaming literacy you would expect at GDC – negotiate and come to grips with the work. (Walter has already talked about the game and the IGF here.)
The game was part of the Independent Game Festival, and set between 2 other games – one was the open-category winner, Savage: Battle for Newerth, a well-funded combination real-time strategy/action game. This meant that the players of the demo had been primed into classic gamer mode. First, they tried to understand the goal of the game; then, to understand the system of operations and the syntax of skilled gameplay needed to navigate that system in order to reach the goals and create other amusing and entertaining effects.
After some time, though, two transitions occurred, the first of which was more basic but more predictable, the second of which was more tentative – and didn’t always occur – but was novel in a very important way.
Transition 1: there is no goal. This is not completely novel: there are plenty of non-game toys or sandboxes within videogames and The Sims have prepared the game audience for this sort of arrangement. Unlike The Sims and most toys, however, the drama management engine pulls the interactions into a story structure that minimizes the toy-like effect – even if there’s no overdetermining telos, there are resolutions and temporal movements, and a player can elect to pursue or avoid one or another resolution. Some players sought to seduce one or both of the characters, or to try to heal the relationship. Another track of play was what I’d call the “nascent alcoholism” track (one player ignored Trip and Grace entirely, and just drank as much SimBooze as possible – the fade-to-black denouement, in his case, was a blackout.) The process of testing the system to discover its boundaries, and the process of simulating socially extreme behavior to do it (kissing one character in the presence of their spouse, flirting with a character who was clearly distraught, using abusive language, etc.), often led to
Transition 2: the agents could make the player feel socially guilty. This was partially an aesthetic effect driven by the interaction between the faciality engine in the character draw routines (as I loosely understand it) with the agents’ internal state systems, and partially by the dramatic context. E.g., when you kissed Grace, especially during an emotionally difficult moment of the drama, her face expressed – in some kind of sequence – effects suggesting confusion, hurt, a little horror, a slight sense of betrayal, quiet reproach, and some social guilt on her own part; her partner, Trip, would also respond in an appropriate way – with a face-saving joke, protest, deeply pained expressions, etc. These effects were the immediate and most straightforward results of player activity, rather than being incidental effects “on the way” to what might have been the player goal (sexual conquest, social dominance, pure troublemaking, etc.).
This has little to do with narrative per se. In a traditional adventure game, the player could be responsible for the death of “innocents” and virtual agents due to clumsy or sadistic performance, but the player does not engage those characters from a substantially interpersonal position. As the player migrates from a mechanistic to a social mode of gameplay, new emotional channels are exposed and accessed.
One felt more responsible and incriminated by hurting Grace’s feelings with inappropriate flirting than one would by killing a Sim, or beating up a young girl in Guilty Gear, or even by perpetrating some atrocity in Knights of the Old Republic, or betraying a character in Deus Ex. This sense of complicity and responsibility was confirmed by severalFacade players I spoke with.
I began to wonder why, and I thought about infancy.
The development of social cognition2 – as distinct from mechanical or spatial or even operational cognition – begins with an early event in most (i.e., non-autistic) infants. Much of the important work on infant social-emotional cognitive development and social cognition has done by John Watson of UC Berkeley and Gyorgy Gergely of University College London. They discovered that infants up to about 3 months in age show a marked and predictable preference for stimuli with perfect contingencies. At the age of 3 months, “normal infants switch to a preference of highly but imperfectly response-contingent stimulation which, in natural circumstances, is provided by responsive social objects.”3 Failure to make this switch is correlated strongly with autism.
What I saw at GDC led me to a clearer idea of what compelling social agents would be, and how they would be part of a meta-genre of game that engaged social, rather than mechanical or systematic, cognition. Social-cognition is more interested in partial contingency: this means agents that are never completely predictable, but are probabilistically predictable. When the player can reduce his/her interaction with the agent to a predictable input/output system – a game of perfect contingency, with or without complete information – players will not bother to engage in social cognition and instead settle into mechanistic, operational play. The problem is that so many interesting mental phenomena occur in social cognition – guilt, identification, personal (rather than managerial) responsibility, love.
Game players are, functionally, autistic during sessions of traditional gameplay, and the effects of the marked centrality of operational/systematic cognition are pervasive enough that they are apparent even when the agents are human – as in MMORPG’s – yet lack the expressive apparata, the embodiedness and smoothly dynamic facialities – to engage the switch from mechanical to social cognition. Even in MMORPG’s, other people/agents are just means to ends, finite-state machines and rational agents appropriate for straightforward interaction for producing results.
Agents that engage social cognition would feature imperfect contingency; also, the faciality engine in Facade was tied into a dynamic state system for the inner lives of the agents. While sorting out – “thinking/reacting” – the player’s activities, the shifting state of the agent was reflected in a series of facial gestures. Any given statement could have a number of responses – generating a certain amount of fear, a bit of jealousy, some anger, some amusement – and the playing out of the “meaning” of the player’s input before it resolves to simple significance (like someone deciding whether or not to laugh along with a perhaps offensive joke, or whether to respond positively to flirtation or not) is visibly apparent to the player, while being patently complex beyond the ability to reduce to a predictable cause/effect relationship.
I’d like to see the narrative vs. gameplay debate reframed. Not again asking games to “tell” better stories, yet without defending the inability to compellingly address questions of the human condition. What I saw with Facade was one way in which games could ultimately surpass other temporal visual media in creating and conveying meaning: by becoming systems that engaged social cognition in ways that involved, rather than just referred to, the sense of place in the world. Traditional media work, perhaps, by recalling our own embodied experience in the world and engaging our imagination accordingly. The lack of interactivity is a prerequisite, not a hindrance, to this sort of creative stance: it is more like a memory being recalled than a process being engaged, and it is a memory that resembles our own social processes.
Game narratives have almost always been primitive and awkward, almost embarrassingly so. The tendencies to epic power fantasies, messianism, and apocalyptic sci-fi themes could be seen as a consequence of the young, largely male audience that is being served, but I think the real issue is structural: independent and experimental game makers have been able to effectively push the aesthetic and thematic components of games despite the relative absence of a supporting market, but efforts to make real headway in terms of narrative and meaning have not paid off.
Instead of trying to just emulate novelistic or filmic recollection/representation, a game-system can stimulate social cognition directly, with agents that imperfectly and unpredictably respond to socially relevant stimuli. Of course, this is in the context of otherwise effective, skillful game design, including the drama management used in Facade to channel the motivations and reactions of the agents into a paced, dynamic narrative. But only with Facade have I seen a real access to new types of player engagement and cognition. More important than including “stories” or even “interesting characters,” is moving past the limitations of a systematized relationship between player and game, and recruiting the deep structures of the player’s mind in the project of using the game to generate real meaning.
 The project page for the game is at InterActiveStory.net. Michael Mateas, one of the developers of Facade, haswritten about believable agents in conjunction with project Oz, the predecessor of Facade.
 This is a good quick overview of the development of social cognition. See also Early Social Cognition, ed. by P Rochat.
 The Obscure Object of Desire: “Nearly, But Clearly Not Like Me” – Perception of Self-Generated Contingencies in Normal Infants and Children with Autism. Gyorgy Gergely, in the Bulletin of the Menninger Clinic, Summer 2001
I’ll write more later, but as you imagine, my time is short.
Posted by: Christian McCrea at March 31, 2004 03:35 PM
Posted by: William at March 31, 2004 03:39 PM
Basically, I’m beginning with classic Campbellian hero function, then going onto rhizomatic heroes and style that destabalise hero narratives, suggest things about agency opposing myth (myth being a key term in the dept) without being drawn into an opinion, hopefully, then finishing on aesthetics and affect; to be attracted to game figures, anticipation of the to-be-done, and the to-be-done-by… which is, I feel, a realm game writers should have long since mined. I end with a fairly dour point about Hans Bellmer’s doll series, and what he said about possession of doll parts, skins to be acted in and through, etc.
Your comments will come in handy near the end to throw up some possibilities for tutorials.
Posted by: Christian McCrea at March 31, 2004 03:46 PM
Sorry, I have less time today than I thought I did. Anyway, just wanted to thank you; great article and the students in my first tutorial of the day got into it, dissecting the value of sci-fi narratives, violence, and via that got into macho-violent consumer fetishism in game culture. I’ll report on the other two if it comes up.
Privately, students have mentioned action sci-fi narratives in games have the scope to deconstruct the generic myths in ways that movies do not; which is interesting to ask – what can game *narratives* provide for cinema (aside from twee crossovers and a focus on visual kinetics, for example). Blade Runner’s PC game was arguably far better scripted than the movie in any case, but the use of branching narratives was sufficiently intelligent to appear entirely fluid. (or rhizomatic, if we dare go there.)
Philosopher Giorgio Agamben’s “Infancy and History” talks about sensory autism quite a bit, I intend to come back to it when this week’s maelstrom settles down.
Sandbox games and simulations are the most profoundly uninteresting games I ever play; I guess that’s why I’ve found a niche to talk about games in a cinema studies department.
I just want to throw some notes up; this isn’t a profound dissection by any means.
Games’ ability to present us with social guilt is immedietely recognisable as a lack. I’ve felt approximations of love, hate.. but rarely guilt. Ico for PS2, though, thrust responsibility into the player role through iconography and audio-visual cues, failure meant a semi-interactive character (Yorda) was doomed.
I guess because I’m interested in the recovery of the terms ‘bravery’ and ‘heroism’ from the deepest recesses of academic thinking, ICO interests me in its presentation of the *approximation* of guilt, bravery, responsibility.
Facade, and games like it, mean something else entirely; where the sandbox concept is extended to emotional figuration. I’ve always felt I’d be a happy gamer when an NPC lies to me, just for the hell of it.
Looking over the GDC, the same debates and faces crop up. Carmack is fascinated by realism and the body, Spector by story. We should perhaps follow the advice we have for games and stop seeking resolutions to essentially irresolute issues?
Posted by: Christian McCrea at March 31, 2004 10:24 PM
Would I be wholly off-base in suggesting that the only difference is one of feedback (game to player) *fidelity* and verisimilitude? I don’t know if I’m comfortable with the idea that a Final Fantasy 6 character doesn’t elicit emotions from me because she’s a 32×32 pixel blob, but a more or less detailed realtime 3D character does. I suppose there *is* a point where fidelity begins to impact the symbolic content of the feedback, though.
“When the player can reduce his/her interaction with the agent to a predictable input/output system – a game of perfect contingency, with or without complete information – players will not bother to engage in social cognition and instead settle into mechanistic, operational play.”
Well said. This is a problem I come back to repeatedly when talking with people about the “emotional involvement in games” issue – you *cannot* reliably say when a player will be playing a game in “socially aware mode”, or what social vector the player will be pursuing (Facade at least seems to do a good job providing for a player who has betrayal, sympathy, romance etc on his agenda) whereas you *can* assume that the operational rapport will be happening (otherwise the player isn’t even touching the controls).
Thus far all of the games I’ve seen that attempt to foster “emotional involvement” fail utterly because they cease to function as games and become a one-way conduit for whatever emotions the designer wants to convey to the player. The usual problems of audience interpretation and agreement (good films certainly don’t demand we agree with them) come into play as well. The autism-like behaviors that players sometimes respond with are, I think, a clear attempt to salvage the operational dimension of interaction, to wrest back control of the discourse as it were.
The challenge is essentially to make a game (or a goal-free “software toy”) out of social interaction, one that plays out on a social level and eschews mechanistic thinking, which would basically have to model social dynamics in the same way that the currently fashionable game physics engines model rigid-body Newtonian physics. That has a lot of interesting implications. I want to see it explored but I’m not sure the advocates of “emotionally involving games” will find it to be the holy grail they are so desperately searching for. At the end of the day, rules are still the language of interactive systems, and many of said advocates are rule-phobic in the extreme (and may flinch at the very idea that social dynamics can be modeled). Progress may be slow.
The alternative, though, is to replace that modeling (process) with a predetermined array of choices (data) as many current games do, which many people will be happy with but that is even more conducive to the dreaded autistic gamesmanship attitude that works so well in games like KotOR.
Interestingly, the only way to truly provide an environment for non-trivial social play still leaves the door open for operational play. I think that’s the consequence of working in an interactive medium.
Posted by: JP at April 1, 2004 09:35 AM
JP: I think you are on half of the right track. The correct part is your assertion that in order for interactions with other agents in v.g.s to be compelling, they have to modelled rather than scripted. Where I think you go off the rails is in presuming that compelling social dynamics preclude scripting, or the even the presence of a designer-implemented *telos*. I would suggest the separation of first-order agent interaction from second-order “narrative” (don’t scream, I used scare quotes) arrangement, something Aarseth’s separation of cybertexts into local and universal patterns of cursality.
There is no reason (excepting the algorithmic complexity involved) that compellingly modelled social agents cannot be embedded in a game such that social interaction becomes one of the many operational strategies available to the player. In fact, I would argue that *this* is the holy grail you mention, as it would cast the pall of social dynamics over all the other available operational possibilities without obviating them entirely. An examples of shuffling toward Bethlehem: the High difficulty level in Thief/II that requires the player not kill in some missions made (at least for me) even a knockout (which, let us note, is mechanically *no different* from a kill) seem like a moral compromise. I realize this isn’t the best example as it reverses the dynamic I proposed immediately before, the affectual implications seem to me to point in the right direction.
Posted by: Ben Buckley at April 5, 2004 09:21 AM
I don’t think it’s a matter of versimilitude, because the agents in Facade aren’t photorealistic – or even particularly detailed in their rendering. However, they have well-articulated, if cartoon-abstracted, features where it matters. In fact, a photo-realistic agent could run into the problem of the uncanny valley, and sabotage the transition to social cognition.
As far as the modelling v. scripting issue goes, Facade is both, to some extent. The utterances are scripted (i.e., not generated) from a large database of possible discourse acts with different tones depending on other questions of agent state (level of agitation, hostility, etc.); the narrative is driven by the drama manager, and the agents are motivated to and away from different dramatic “beats.” This is not a case of winding up socially-compelling agents and letting the narrative just “fall out;” but the narrative itself is almost like a Rube Goldberg machine, created by the player’s interaction with a “Rube Goldberg meta-machine” that includes these socially responsive, social-cognition compelling, motivated agents.
Ben, I think it’s already easy to use a kind of mechanistic social information in games. “The guard is afraid of snakes. I have a snake. I will scare the guard with the snake, he will run away and I will grab the secret documents.” But I actually think that is part of the problematic turn towards the operational stance (Heideggerian “being-at-hand”; agents/people in standing reserve) – what I’m talking about is actually using social cognition to handle social problem “I feel guilty, which makes me realize that I am capable of hurting the “feelings” of this agent; in order to avoid guilt, I need to say something reassuring without being unbelievable; she seems repelled and nervous, which means that she’s not comfortable with my advances and that she has some loyalty to her marriage still.” There are complex state systems in the agents which we interact with using the most basic interface we have: the interpretation of facial expressions, to which a considerable amount of cortical “real estate” is dedicated.
Posted by: William at April 5, 2004 10:37 AM
But you’ve made we curious. Perhaps I misunderstood what you meant by “operational stance,” because when I read that, it had nothing to do with Heideger. Have I misread, or could you expand?
Posted by: Ben B. at April 5, 2004 03:14 PM
But as much as I enjoy bringing Heidegger into the conversation, I wouldn’t want to spend too much time there: the purpose is to identify the problem, not to sing to it.
Posted by: William at April 5, 2004 03:20 PM