I’m at a talk that Gonzalo Frasca just gave on game rhetoric and the role of performance. (Takeaway phrase: “haptic semiotics.” Another take on the embodiment/”chaism”/activity theory/Heideggerian turn in new media and HCI theory, but a happy phrase nonetheless.) But it has prompted a backchannel discussion on hardcore gamers, gamer identity, etc.

The contradictory elements of this discourse are worth observing. Many gamers I have met and spoken with have long expressed the hope that videogaming be recognized as a mainstream practice. Now that it is, of course, they seek to distinguish their own mode of reception from those of the new demographics, who bring their own needs, expectations, and constraints to the constellation of mediating, representing, and activity-structuring elements from videogames. In many ways, it is the casual gamers which will broaden the expressive and rhetorical range of game design, rather than the traditional players.

I wonder how long the anxiety of being a “true gamer” will last; at one time, literacy was exceptional, rare, unusual. The ability to read and write identified one as a member of an elite. There is no longer a demographic or caste of “readers.” And the best of written material created for a widely literate public (i.e., the novel) would have been impossible to create if literacy had been constrained to a class of elite scribes.

Building Projection Round-Up

Genesis Tower
Originally uploaded by ThisDude62

Via Penny Arcade, a look at the madness that is expandability: the Sega Genesis and its augmentations.

Theatre of the Oppressed Founder Augusto Boal Dies at 78: Theater News on TheaterMania.com – 05/02/2009

I think that Boal’s significance for interactive design has yet to be taken in fully. Gonzalo Frasca’s “Videogames of the Oppressed describes how his ideas might be taken up by videogame design. What is still hard to capture is the dialogic nature of Boal’s work; perhaps Brecht is still a more viable model for the videogame engagée. Distanciation without apathy (or instrumentalization) is a difficult enough principle to work from.

In any case, farewell, Augusto Boal.

I’ve completed my prospectus, met with my committee: I’m now a doctoral candidate, in the limbo that is ABD.

My research proposal can be summed up this way: when cultural artifacts are built of software, what is the nature and significance of software engineering as a form of collective and collaborative authorship (or artistic practice?) I’m planning to look at a range of work, from very commercial videogames to open-source projects to art-games made for installations.

I’ve completed my prospectus, met with my committee: I’m now a doctoral candidate, in the limbo that is ABD.

My research proposal can be summed up this way: when cultural artifacts are built of software, what is the nature and significance of software engineering as a form of collective and collaborative authorship (or artistic practice?) I’m planning to look at a range of work, from very commercial videogames to open-source projects to art-games made for installations.

Unsurprisingly, my interest in the aesthetic has more to do with Adorno that it has to do with ray-tracing. I’ve generally been disinterested in the problems of “realism” and 3D-graphics above and beyond the general question of producing  experiences of navigable spaces and introducing the various ways of thinking about space that might emerge from them.

I am, however, becoming more interested in the technical problems, and seeing in them (like Panofsky found in perspective in painting) substantial aesthetic and conceptual issues of their own. This interview with Jay Wilsonthe lead designer of Diablo III shows some of the issues in play. Some players responded to screenshots from the game that they found disappointing (for the most part, because they didn’t seem to satisfy the players’ desire for over-the-top goth-metal excess); Wilson notes the difference between rendering still images in Photoshop and rendering dynamic scenes in 3D, as well as addressing questions of visual aesthetic taste and variety.

Unsurprisingly, my interest in the aesthetic has more to do with Adorno that it has to do with ray-tracing. I’ve generally been disinterested in the problems of “realism” and 3D-graphics above and beyond the general question of producing  experiences of navigable spaces and introducing the various ways of thinking about space that might emerge from them.

I am, however, becoming more interested in the technical problems, and seeing in them (like Panofsky found in perspective in painting) substantial aesthetic and conceptual issues of their own. This interview with Jay Wilsonthe lead designer of Diablo III shows some of the issues in play. Some players responded to screenshots from the game that they found disappointing (for the most part, because they didn’t seem to satisfy the players’ desire for over-the-top goth-metal excess); Wilson notes the difference between rendering still images in Photoshop and rendering dynamic scenes in 3D, as well as addressing questions of visual aesthetic taste and variety.

The following is a copy of a post I made to the defunct and currently-unavailable Ludonauts group blog. The original URL was http:/www.ludonauts.com/archives/000038.shtml; I’m also copying the comments. It goes without saying, I would probably write this differently today (mostly by being a little less confident about the narrative of social cognitive development – there are other theories, and the one that I deployed here shouldn’t be read as having universal acceptance.)

The original date of this post was March 30, 2004

…

I recently had the opportunity to help the creators of Facade demo their game/drama¹ at the 2004 GDC. What I saw, when watching people play it, brought into focus for me just why I have been pessimistic about games as a compelling media for work that is as rich in meaning and humanistic depth as film and literature can be; and I think I understand what will have to happen before what we now call “videogames” evolves into something that will become the definitive expressive practice for the next century. It does – and does not – involve the old rhubarbs “narrative” and “character” and “story.” Instead, it’s about cognition – and a shift in just what cognitive fluencies are engaged by the game to navigate game-space.

In Facade, the player has been invited to the upscale apartment of a married couple, Trip and Grace – old college friends whom the player had introduced to each other 10 years ago – for drinks. It quickly becomes apparent that their marriage is in crisis, and the interaction of the player with the characters moves the plot along in a way entirely unlike the typical “choose your own adventure” way. The engine(s) behind the game are interesting enough – there is a semantic parser which passes the player’s statements and actions to the AI’s, which in turn interact with a drama manager that moves the situation and conversations to distinct dramatic “beats.” The interchange between these motivated AI agents and the structures of the drama manager is a compelling resolution of the tension between character-driven and narrative-driven interactive storytelling, but what actually caught my eye was watching players – gamers, most of them, with the expectations and gaming literacy you would expect at GDC – negotiate and come to grips with the work. (Walter has already talked about the game and the IGF here.)

The game was part of the Independent Game Festival, and set between 2 other games – one was the open-category winner, Savage: Battle for Newerth, a well-funded combination real-time strategy/action game. This meant that the players of the demo had been primed into classic gamer mode. First, they tried to understand the goal of the game; then, to understand the system of operations and the syntax of skilled gameplay needed to navigate that system in order to reach the goals and create other amusing and entertaining effects.

After some time, though, two transitions occurred, the first of which was more basic but more predictable, the second of which was more tentative – and didn’t always occur – but was novel in a very important way.

Transition 1: there is no goal. This is not completely novel: there are plenty of non-game toys or sandboxes within videogames and The Sims have prepared the game audience for this sort of arrangement. Unlike The Sims and most toys, however, the drama management engine pulls the interactions into a story structure that minimizes the toy-like effect – even if there’s no overdetermining telos, there are resolutions and temporal movements, and a player can elect to pursue or avoid one or another resolution. Some players sought to seduce one or both of the characters, or to try to heal the relationship. Another track of play was what I’d call the “nascent alcoholism” track (one player ignored Trip and Grace entirely, and just drank as much SimBooze as possible – the fade-to-black denouement, in his case, was a blackout.) The process of testing the system to discover its boundaries, and the process of simulating socially extreme behavior to do it (kissing one character in the presence of their spouse, flirting with a character who was clearly distraught, using abusive language, etc.), often led to

Transition 2: the agents could make the player feel socially guilty. This was partially an aesthetic effect driven by the interaction between the faciality engine in the character draw routines (as I loosely understand it) with the agents’ internal state systems, and partially by the dramatic context. E.g., when you kissed Grace, especially during an emotionally difficult moment of the drama, her face expressed – in some kind of sequence – effects suggesting confusion, hurt, a little horror, a slight sense of betrayal, quiet reproach, and some social guilt on her own part; her partner, Trip, would also respond in an appropriate way – with a face-saving joke, protest, deeply pained expressions, etc. These effects were the immediate and most straightforward results of player activity, rather than being incidental effects “on the way” to what might have been the player goal (sexual conquest, social dominance, pure troublemaking, etc.).

This has little to do with narrative per se. In a traditional adventure game, the player could be responsible for the death of “innocents” and virtual agents due to clumsy or sadistic performance, but the player does not engage those characters from a substantially interpersonal position. As the player migrates from a mechanistic to a social mode of gameplay, new emotional channels are exposed and accessed.

One felt more responsible and incriminated by hurting Grace’s feelings with inappropriate flirting than one would by killing a Sim, or beating up a young girl in Guilty Gear, or even by perpetrating some atrocity in Knights of the Old Republic, or betraying a character in Deus Ex. This sense of complicity and responsibility was confirmed by severalFacade players I spoke with.

I began to wonder why, and I thought about infancy.

The development of social cognition² – as distinct from mechanical or spatial or even operational cognition – begins with an early event in most (i.e., non-autistic) infants. Much of the important work on infant social-emotional cognitive development and social cognition has done by John Watson of UC Berkeley and Gyorgy Gergely of University College London. They discovered that infants up to about 3 months in age show a marked and predictable preference for stimuli with perfect contingencies. At the age of 3 months, “normal infants switch to a preference of highly but imperfectly response-contingent stimulation which, in natural circumstances, is provided by responsive social objects.”³ Failure to make this switch is correlated strongly with autism.

What I saw at GDC led me to a clearer idea of what compelling social agents would be, and how they would be part of a meta-genre of game that engaged social, rather than mechanical or systematic, cognition. Social-cognition is more interested in partial contingency: this means agents that are never completely predictable, but are probabilistically predictable. When the player can reduce his/her interaction with the agent to a predictable input/output system – a game of perfect contingency, with or without complete information – players will not bother to engage in social cognition and instead settle into mechanistic, operational play. The problem is that so many interesting mental phenomena occur in social cognition – guilt, identification, personal (rather than managerial) responsibility, love.

Game players are, functionally, autistic during sessions of traditional gameplay, and the effects of the marked centrality of operational/systematic cognition are pervasive enough that they are apparent even when the agents are human – as in MMORPG’s – yet lack the expressive apparata, the embodiedness and smoothly dynamic facialities – to engage the switch from mechanical to social cognition. Even in MMORPG’s, other people/agents are just means to ends, finite-state machines and rational agents appropriate for straightforward interaction for producing results.

Agents that engage social cognition would feature imperfect contingency; also, the faciality engine in Facade was tied into a dynamic state system for the inner lives of the agents. While sorting out – “thinking/reacting” – the player’s activities, the shifting state of the agent was reflected in a series of facial gestures. Any given statement could have a number of responses – generating a certain amount of fear, a bit of jealousy, some anger, some amusement – and the playing out of the “meaning” of the player’s input before it resolves to simple significance (like someone deciding whether or not to laugh along with a perhaps offensive joke, or whether to respond positively to flirtation or not) is visibly apparent to the player, while being patently complex beyond the ability to reduce to a predictable cause/effect relationship.

I’d like to see the narrative vs. gameplay debate reframed. Not again asking games to “tell” better stories, yet without defending the inability to compellingly address questions of the human condition. What I saw with Facade was one way in which games could ultimately surpass other temporal visual media in creating and conveying meaning: by becoming systems that engaged social cognition in ways that involved, rather than just referred to, the sense of place in the world. Traditional media work, perhaps, by recalling our own embodied experience in the world and engaging our imagination accordingly. The lack of interactivity is a prerequisite, not a hindrance, to this sort of creative stance: it is more like a memory being recalled than a process being engaged, and it is a memory that resembles our own social processes.

Game narratives have almost always been primitive and awkward, almost embarrassingly so. The tendencies to epic power fantasies, messianism, and apocalyptic sci-fi themes could be seen as a consequence of the young, largely male audience that is being served, but I think the real issue is structural: independent and experimental game makers have been able to effectively push the aesthetic and thematic components of games despite the relative absence of a supporting market, but efforts to make real headway in terms of narrative and meaning have not paid off.

Instead of trying to just emulate novelistic or filmic recollection/representation, a game-system can stimulate social cognition directly, with agents that imperfectly and unpredictably respond to socially relevant stimuli. Of course, this is in the context of otherwise effective, skillful game design, including the drama management used in Facade to channel the motivations and reactions of the agents into a paced, dynamic narrative. But only with Facade have I seen a real access to new types of player engagement and cognition. More important than including “stories” or even “interesting characters,” is moving past the limitations of a systematized relationship between player and game, and recruiting the deep structures of the player’s mind in the project of using the game to generate real meaning.

Notes:

[1] The project page for the game is at InterActiveStory.net. Michael Mateas, one of the developers of Facade, haswritten about believable agents in conjunction with project Oz, the predecessor of Facade.

[2] This is a good quick overview of the development of social cognition. See also Early Social Cognition, ed. by P Rochat.

[3] The Obscure Object of Desire: “Nearly, But Clearly Not Like Me” – Perception of Self-Generated Contingencies in Normal Infants and Children with Autism. Gyorgy Gergely, in the Bulletin of the Menninger Clinic, Summer 2001

The following is a copy of a post I made to the defunct and currently-unavailable Ludonauts group blog. The original URL was http:/www.ludonauts.com/archives/000038.shtml; I’m also copying the comments. It goes without saying, I would probably write this differently today (mostly by being a little less confident about the narrative of social cognitive development – there are other theories, and the one that I deployed here shouldn’t be read as having universal acceptance.)

The original date of this post was March 30, 2004

…

I recently had the opportunity to help the creators of Facade demo their game/drama¹ at the 2004 GDC. What I saw, when watching people play it, brought into focus for me just why I have been pessimistic about games as a compelling media for work that is as rich in meaning and humanistic depth as film and literature can be; and I think I understand what will have to happen before what we now call “videogames” evolves into something that will become the definitive expressive practice for the next century. It does – and does not – involve the old rhubarbs “narrative” and “character” and “story.” Instead, it’s about cognition – and a shift in just what cognitive fluencies are engaged by the game to navigate game-space.

In Facade, the player has been invited to the upscale apartment of a married couple, Trip and Grace – old college friends whom the player had introduced to each other 10 years ago – for drinks. It quickly becomes apparent that their marriage is in crisis, and the interaction of the player with the characters moves the plot along in a way entirely unlike the typical “choose your own adventure” way. The engine(s) behind the game are interesting enough – there is a semantic parser which passes the player’s statements and actions to the AI’s, which in turn interact with a drama manager that moves the situation and conversations to distinct dramatic “beats.” The interchange between these motivated AI agents and the structures of the drama manager is a compelling resolution of the tension between character-driven and narrative-driven interactive storytelling, but what actually caught my eye was watching players – gamers, most of them, with the expectations and gaming literacy you would expect at GDC – negotiate and come to grips with the work. (Walter has already talked about the game and the IGF here.)

The game was part of the Independent Game Festival, and set between 2 other games – one was the open-category winner, Savage: Battle for Newerth, a well-funded combination real-time strategy/action game. This meant that the players of the demo had been primed into classic gamer mode. First, they tried to understand the goal of the game; then, to understand the system of operations and the syntax of skilled gameplay needed to navigate that system in order to reach the goals and create other amusing and entertaining effects.

After some time, though, two transitions occurred, the first of which was more basic but more predictable, the second of which was more tentative – and didn’t always occur – but was novel in a very important way.

Transition 1: there is no goal. This is not completely novel: there are plenty of non-game toys or sandboxes within videogames and The Sims have prepared the game audience for this sort of arrangement. Unlike The Sims and most toys, however, the drama management engine pulls the interactions into a story structure that minimizes the toy-like effect – even if there’s no overdetermining telos, there are resolutions and temporal movements, and a player can elect to pursue or avoid one or another resolution. Some players sought to seduce one or both of the characters, or to try to heal the relationship. Another track of play was what I’d call the “nascent alcoholism” track (one player ignored Trip and Grace entirely, and just drank as much SimBooze as possible – the fade-to-black denouement, in his case, was a blackout.) The process of testing the system to discover its boundaries, and the process of simulating socially extreme behavior to do it (kissing one character in the presence of their spouse, flirting with a character who was clearly distraught, using abusive language, etc.), often led to

Transition 2: the agents could make the player feel socially guilty. This was partially an aesthetic effect driven by the interaction between the faciality engine in the character draw routines (as I loosely understand it) with the agents’ internal state systems, and partially by the dramatic context. E.g., when you kissed Grace, especially during an emotionally difficult moment of the drama, her face expressed – in some kind of sequence – effects suggesting confusion, hurt, a little horror, a slight sense of betrayal, quiet reproach, and some social guilt on her own part; her partner, Trip, would also respond in an appropriate way – with a face-saving joke, protest, deeply pained expressions, etc. These effects were the immediate and most straightforward results of player activity, rather than being incidental effects “on the way” to what might have been the player goal (sexual conquest, social dominance, pure troublemaking, etc.).

This has little to do with narrative per se. In a traditional adventure game, the player could be responsible for the death of “innocents” and virtual agents due to clumsy or sadistic performance, but the player does not engage those characters from a substantially interpersonal position. As the player migrates from a mechanistic to a social mode of gameplay, new emotional channels are exposed and accessed.

One felt more responsible and incriminated by hurting Grace’s feelings with inappropriate flirting than one would by killing a Sim, or beating up a young girl in Guilty Gear, or even by perpetrating some atrocity in Knights of the Old Republic, or betraying a character in Deus Ex. This sense of complicity and responsibility was confirmed by severalFacade players I spoke with.

I began to wonder why, and I thought about infancy.

The development of social cognition² – as distinct from mechanical or spatial or even operational cognition – begins with an early event in most (i.e., non-autistic) infants. Much of the important work on infant social-emotional cognitive development and social cognition has done by John Watson of UC Berkeley and Gyorgy Gergely of University College London. They discovered that infants up to about 3 months in age show a marked and predictable preference for stimuli with perfect contingencies. At the age of 3 months, “normal infants switch to a preference of highly but imperfectly response-contingent stimulation which, in natural circumstances, is provided by responsive social objects.”³ Failure to make this switch is correlated strongly with autism.

What I saw at GDC led me to a clearer idea of what compelling social agents would be, and how they would be part of a meta-genre of game that engaged social, rather than mechanical or systematic, cognition. Social-cognition is more interested in partial contingency: this means agents that are never completely predictable, but are probabilistically predictable. When the player can reduce his/her interaction with the agent to a predictable input/output system – a game of perfect contingency, with or without complete information – players will not bother to engage in social cognition and instead settle into mechanistic, operational play. The problem is that so many interesting mental phenomena occur in social cognition – guilt, identification, personal (rather than managerial) responsibility, love.

Game players are, functionally, autistic during sessions of traditional gameplay, and the effects of the marked centrality of operational/systematic cognition are pervasive enough that they are apparent even when the agents are human – as in MMORPG’s – yet lack the expressive apparata, the embodiedness and smoothly dynamic facialities – to engage the switch from mechanical to social cognition. Even in MMORPG’s, other people/agents are just means to ends, finite-state machines and rational agents appropriate for straightforward interaction for producing results.

Agents that engage social cognition would feature imperfect contingency; also, the faciality engine in Facade was tied into a dynamic state system for the inner lives of the agents. While sorting out – “thinking/reacting” – the player’s activities, the shifting state of the agent was reflected in a series of facial gestures. Any given statement could have a number of responses – generating a certain amount of fear, a bit of jealousy, some anger, some amusement – and the playing out of the “meaning” of the player’s input before it resolves to simple significance (like someone deciding whether or not to laugh along with a perhaps offensive joke, or whether to respond positively to flirtation or not) is visibly apparent to the player, while being patently complex beyond the ability to reduce to a predictable cause/effect relationship.

I’d like to see the narrative vs. gameplay debate reframed. Not again asking games to “tell” better stories, yet without defending the inability to compellingly address questions of the human condition. What I saw with Facade was one way in which games could ultimately surpass other temporal visual media in creating and conveying meaning: by becoming systems that engaged social cognition in ways that involved, rather than just referred to, the sense of place in the world. Traditional media work, perhaps, by recalling our own embodied experience in the world and engaging our imagination accordingly. The lack of interactivity is a prerequisite, not a hindrance, to this sort of creative stance: it is more like a memory being recalled than a process being engaged, and it is a memory that resembles our own social processes.

Game narratives have almost always been primitive and awkward, almost embarrassingly so. The tendencies to epic power fantasies, messianism, and apocalyptic sci-fi themes could be seen as a consequence of the young, largely male audience that is being served, but I think the real issue is structural: independent and experimental game makers have been able to effectively push the aesthetic and thematic components of games despite the relative absence of a supporting market, but efforts to make real headway in terms of narrative and meaning have not paid off.

Instead of trying to just emulate novelistic or filmic recollection/representation, a game-system can stimulate social cognition directly, with agents that imperfectly and unpredictably respond to socially relevant stimuli. Of course, this is in the context of otherwise effective, skillful game design, including the drama management used in Facade to channel the motivations and reactions of the agents into a paced, dynamic narrative. But only with Facade have I seen a real access to new types of player engagement and cognition. More important than including “stories” or even “interesting characters,” is moving past the limitations of a systematized relationship between player and game, and recruiting the deep structures of the player’s mind in the project of using the game to generate real meaning.

Notes:

[1] The project page for the game is at InterActiveStory.net. Michael Mateas, one of the developers of Facade, haswritten about believable agents in conjunction with project Oz, the predecessor of Facade.

[2] This is a good quick overview of the development of social cognition. See also Early Social Cognition, ed. by P Rochat.

[3] The Obscure Object of Desire: “Nearly, But Clearly Not Like Me” – Perception of Self-Generated Contingencies in Normal Infants and Children with Autism. Gyorgy Gergely, in the Bulletin of the Menninger Clinic, Summer 2001