Judy Malloy, Editor

D. Fox Harrell
The GRIOT System

Figure 1: D. Fox Harrell: The GRIOT System Architecture

R esearcher, writer and artist, D. Fox Harrell is Associate Professor of Digital Media in the Comparative Media Studies Program and Computer Science and Artificial Intelligence Laboratory at MIT. Exploring the relationship between imaginative cognition and computation, Harrell's work focuses on the development of computer-media-narrative and authoring software that use elements of interactivity, social critique, cross-cultural narrative; cognitive semantics; gaming; and the social aspects of user-interface design.

His seminal GRIOT System utilizes code to create/generate interactive and significant "polymorphic" poems -- such as The Girl with Skin of Haints and Seraphs and Walking Blues Changes Undersea. GRIOT (named for West African storytellers who often incorporate improvisation in their performances) uses a combination of knowledge engineering, interactivity, cultural identity, and Joseph Goguen's mathematical approach to meaning representation called algebraic semiotics. Harrell has also worked with Kenny Chow to create a "new form of concrete polymorphic poetry inspired by Japanese renku poetry, iconicity of Chinese character forms, and generative models from contemporary art." [1]

D. Fox Harrell's work continues to provide a basis for interactive and generative multimedia systems. In his book chapter for Second Person [2] Harrell has stated that his "longer-term project involves using this technical and theoretical framework as a basis for creating further computational narrative artworks where in addition to textual input, users can interact with graphical or gamelike interfaces. This user interaction will still drive the generation of new metaphors and concepts, but along with text will also result in blends of graphical and/or audio media." Indeed, his recent projects such as Living Liberia Fabric, an interactive narrative peace memorial affiliated with the Truth and Reconciliation of Liberia, have fulfilled this early objective.

Fox Harrell's work has been published, performed and exhibited internationally, including the University of Toronto Press, MIT Press, Elsevier, Springer-Verlag, CTheory, The Journal of Research into New Media Technologies, The Fibreculture Journal, and the Electronic Literature Organization, and he is the recipient of the National Science Foundation CAREER Award for his project "Computing for Advanced Identity Representation." His book, Phantasmal Media: An Approach to Imagination, Computation, and Expression was published by MIT Press in 2013.

1.D. Fox Harrell and Kenny K. N. Chow
"Generative Visual Renku: Poetic Multimedia Semantics with the GRIOT System"
HYPERRHIZ.06, Special Issue: Visionary Landscapes, Summer 2008
2. D. Fox Harrell, "GRIOT's Tales of Haints and Seraphs, A Computational Narrative Generation System" in Pat Harrigan and Noah Wardrip-Fruin, eds., Second Person Role-Playing and Story in Games and Playable Media, MIT Press, 2010

Fox Harrell: The GRIOT System

T he GRIOT system was invented by Fox Harrell, and it comprises technical support for implementing narrative and other forms of computational discourse with the following characteristics: generative content, semantics-based interaction, reconfigurable discourse structure, and strong cognitive and socio-cultural grounding. Strong cognitive and socio-cultural grounding here implies that meaning is considered to be contextual, dynamic, and embodied. The formalizations used derive from cognitive linguistics theories with such notions of meaning. Furthermore, the notion of narrative here is not biased toward one particular cultural model. Using semantically based media elements as a foundation, an author can implement a range of culturally specific or experimental narrative structures.

The GRIOT Architecture

The following describes the GRIOT architecture as used in initial text-based experiments with narratively structured poetry. User input, in the form of keywords, is used to select the conceptual space network from a set of ontologies, called "theme domains," that each contain sets of axioms about a particular theme. These axioms consist of binary relations between sorted constants. This conceptual space network, called an "input diagram," consists of a generic space, two input spaces, and mappings from the generic space to each of the input spaces. The input diagram is passed as input to the ALLOY conceptual blending algorithm.

ALLOY is the core component of GRIOT that is responsible for generating new content. An "output diagram," consisting of a blended conceptual space and morphisms from the input spaces to the blended space, is output by ALLOY. Concepts are combined according to principles that produce "optimal" blends. Typically this optimality results in "common sense" blends, but for particular poetic effects different, "dis-optimal" criteria can be utilized.

"Phrase templates," granular fragments of poetry organized by narrative clause type, are combined with the output of ALLOY (converted to natural language by mappings called "grammar morphisms") to result in poems that differ not only in how the phrases are selected and configured, but in the meaning being expressed by the blended concepts.

The phrases are said to be "instantiated" when they are combined with the natural language representations of the blends by replacing "wildcards" in the text. These wildcards are tokens representing where generated output can be incorporated; they also contain variables that specify how they are to be replaced, e.g. constraining the choice of theme domains, or selecting the lexical form to be mapped to by the grammar morphism. These templates are selected according to an automaton called a "Narrative Structure Machine," which also structures the reading of user input.

Multimedia Semantics
The GRIOT system can also dynamically compose modular graphical elements. This functionality is split between a server (implemented in LISP) to handle semantics and graphical discourse structuring and a client (implemented in Processing) to handle graphics processing and user input as depicted in the Figure below.

Figure 2: The Generative GRIOT Multimedia Semantics System

The server consists of the following components:

1) Semantic annotation
2] Discourse structuring rules
3) Priority morphism (matching) algorithm
4) Image layout data structure

This extends elements of the GRIOT system with an algorithm to judge fitness (matching) of images to be composed and a layout structure for multimedia images in addition to the standard discourse structuring rules. The client consists of the following components:

5) Graphical assets
6) Graphical layout rules
7) Duplicate image layout data structure
8) GUI input and output protocols

Graphical assets are actual image data files. These assets are described on the server side using semantic annotation. (the relationship between the metadata and image data files is indicated by a dotted line in the Multimedia Semantics Figure above) This annotation, using XML that is parsed and processed to produce LISP data structures, describes the visual, structural, and conceptual content of images.

Fox Harrell's statement on The Griot System originally appeared on the website of the Digital Media Program, School of Literature, Communications, and Culture Georgia Institute of Technology

Authoring Software


Writers and Artists
Talk about Their Work
and the Software They
use to Create Their Work

Mark Amerika

Mark Bernstein
__Interview with Mark Bernstein

Bill Bly

Jay Bushman

J. R. Carpenter
__ The Broadside of a Yarn
__ Entre Ville
__ Chronicles of Pookie and JR

M.D. Coverley
__ Egypt: The Book of
Going Forth by Day

__ Tin Towns

Caitlin Fisher

Chris Funkhouser

Dene Grigar
__ 24-Hr. Micro-Elit
__ Fallow Field

Fox Harrell

William Harris

Megan Heyward

Adriene Jenik

Antoinette LaFarge

Deena Larsen
__ Marble Springs 3.0
__ The Pines at Walden Pond

Judy Malloy

Mark C. Marino

María Mencía

Nick Montfort
__Nick Montfort and
Stephanie Strickland
Sea and Spar Between

Judd Morrissey

Stuart Moulthrop
__ Interview with
Stuart Moulthrop

Karen O'Rourke

Regina Pinto

Andrew Plotkin
__ Hoist Sail for the Heliopause and Home
__ The Dreamhold

Sonya Rapoport:
__Interview with
Sonya Rapoport

Aaron Reed

Scott Rettberg

Silvia Stoyanova and Ben Johnston
The Zibaldone Hypertext Research Platform

Stephanie Strickland
__Nick Montfort and Stephanie Strickland
Sea and Spar Between

Eugenio Tisselli

Dan Waber