John Lee
Human Communication Research Centre
and EdCAAD, Dept. of Architecture
University of Edinburgh
2 Buccleuch Place
Edinburgh EH8 9LW
The locus classicus of comparative study between graphical and linguistic systems is Nelson Goodman's Languages of Art (1968). Goodman is concerned with a general issue about how representation works -- how marks on paper are related to various kinds of things in the world[1] -- in a range of cases such as pictures, music and other kinds of notation. He therefore seeks to establish what distinguishes a "notational symbol system" from other kinds of symbol system. His approach forms the prototype for most later formal theories in this area, in as much as he considers even pictures to be symbol systems which represent not in virtue of any notion such as resemblance, but due to their being subject to certain systematic rules of use.
According to Goodman, there are five basic conditions required for a symbol system to be notational.
1. It must consist of symbols (utterances, inscriptions, marks) which form equivalence classes (characters) on the basis that they can be exchanged without syntactical effect. Alphabets are a prototypical example -- any "a" is as good as any other; they are "character-indifferent", and the characters have to be disjoint, so that no mark qualifies as an instance of more than one character. In general, Goodman takes compound inscriptions (e.g. sentences) to be characters as well.
2. Characters have to be "finitely differentiable" (or "articulate") in the sense that their disjointness is feasibly testable, which rules out e.g. "dense" systems where any two (ordered) characters have another between them.
3. Notational systems must be unambiguous, so that the extension (which Goodman calls the "compliance-class") of an inscription is invariant with respect to time, context, etc.
4. The compliance-classes of all characters must be disjoint. (Also, the system will ideally be non-redundant.)
5. Compliance-classes must also be finitely differentiable. Thus, for example, any system which is "semantically dense", in that its compliants form an ordering such that any two have another between them, is excluded.
Goodman elaborates these points in relation to clocks and pressure gauges. These measure quantities that are infinitely variable, so that the semantic domain can always be seen as dense, and if there are no marks on the dial, then there is no syntactic differentiation of characters, so the representation system is clearly non-notational. It can become syntactically notational if, say, dots are distributed around the dial and each is taken to be the centre of a disjoint region such that the pointer appearing anywhere within that region counts as an inscription of a certain character. If the ranges of pressure correlated with these regions are also disjoint (and articulate, of course), then the system meets the semantic requirements as well, and hence is simply a notation. On an ordinary clock face, the hour hand is typically used notationally in this way, whereas the minute hand may be seen as marking the absolute elapsed time since the passing of a particular mark, and hence is non-notational.
Diagrams, one might think, are typically non-notational. Goodman observes that many topological diagrams are in fact entirely notational. This also applies e.g. to many drawings used in architecture and design, where although there may be a non-notational impression of form, measurements etc. are always given and the use of the drawing becomes mainly notational. Ordinary road maps are a typical example of mixed diagrams, with both notational and non-notational aspects. Where non-notational, Goodman observes, diagrams are equivalent to two-dimensional models, if we narrow the latter term (which in general can mean "almost anything from a naked blonde to a quadratic equation") so that it excludes descriptions and samples. Models or diagrams of molecules, say, are usually entirely notational; others range all the way to being entirely non-notational.
Goodman approaches the difference between diagrams and pictures by introducing a further notion of "repleteness". A symbol is relatively replete if a relatively large number of its properties are involved in its identity as a symbol; something is more a picture, and less a mere diagram, if there is less about it that can be changed without making it into a different picture. This concept recieves more detailed discussion below.
Goodman's general view is summarised as follows:
Descriptions are distinguished from depictions not through being more arbitrary but through belonging to articulate rather than to dense schemes; and words are more conventional than pictures only if convention is construed in terms of differentiation rather than of artificiality. (230-231)
According to his own account, however, Goodman is not here trying to define the pictorial. Writing much later, in "Representation re-presented" (Goodman & Elgin 1988, ch. VIII), he says:
Nowhere in my writing to date have I proposed a definition of depiction, but have only suggested that the everyday classification of symbols into pictures and nonpictures is related in an important way to the line between symbols in a dense or 'analog' system and those in a finitely differentiated or 'digital' system. Being analog seems perhaps a necessary condition for a depiction, but not a sufficient one; an ungraduated thermometer indicates temperatures in an analog system but does not, in common parlance, picture them. (Goodman & Elgin op. cit., 123)
This characterisation is then sharpened up somewhat by noting that the distinction between analog and digital does not depend on the semantics of the system. Considering only the syntactic aspect (called a scheme, where clearly a scheme, being susceptible of having different denotations assigned to it, can belong to more than one system), Goodman notes that digital and analog schemes can be categorised on the basis of differentiation among the symbols in the scheme. Goodman is thus led to claim that the pictorial can be distinguished from the verbal on a purely syntactic basis, despite the apparently paradoxical facts that "all symbols belong to many digital and analog schemes", and "some schemes consisting entirely of pictures ... are digital" (130). The key to resolving this paradox is that one has to consider the comprehensive or full scheme for a whole language (e.g. English) or pictorial system (e.g. our presystematic notion of what a picture is).
This vehement argument begins to call into question the very existence, or at least definiteness, of the system of rules; the syntax and semantics. The identification -- and hence identity -- of a word, or its location in a grammatical category, becomes open to question. If we look back at Goodman's approach to defining a syntax, we note that it depends on discriminable marks that fall into equivalence classes and are interpreted unambiguously. In fact, few symbol systems in practical use will meet these criteria, and the observations in the previous paragraph serve to emphasise that even when they may appear to this is likely to be an illusion. How, in fact, are the relevant equivalence classes identified? -- By the patterns of use that the symbols are subject to, e.g. what can be exchanged "without syntactical effect". But such effects can only be identified on the basis of a certain amount of theorising, which in generating the distinction between syntax and semantics (and that which is neither) departs from the reality of practice where context and relation to experience are everything. Any distinction so generated is surely to be regarded as bounded and perhaps temporary, certainly subject to revision in the face of different kinds of usage.
In these circumstances, can we really speak of a comprehensive symbol scheme? Difficult as this must be for the symbols of a language, it seems still more so for those constituting a pictorial system. As Goodman himself emphasises, one and the same picture may appear in one situation as a digital character, in another as an analog picture. It seems manifestly implausible that we can tell which is which on purely syntactic grounds, because this requires us to establish when the picture can be substituted by another; and even if this can be found out from an agnostic scrutiny of patterns of usage, it surely still depends on what the picture is taken to represent. On the one hand, it is deeply problematic to identify the system that is at hand when any symbol is being considered; on the other hand, as far as pictures are concerned it appears that when used analogically each is a unique exemplar of a symbol and hence, as Elkins (1993) observes, that "there is very little sense in calling non-notational images 'systems'" (361).
A defence of the syntactic approach is mounted by Scholz (1993, 101-2) on the basis that pictures are common enough which do not denote at all -- e.g. pictures of fictional objects. We can accept this without finding it very helpful. In all symbol systems there's a sense in which what something means is distinct from the question of whether anything corresponds to this. Elgin (1993, 135), responding to Scholz, makes a similar point in observing that reference, as understood by herself and Goodman, encompasses more than denotation, including e.g. exemplification, expression and allusion. For these or other reasons, we surely have to insist that symbols which fail to denote "real world" objects are not thereby shown to lack interesting semantic properties; but also it is hard to see that syntactic properties alone can be enough to distinguish pictures from other symbols.
Goodman worries that
The pictorial is distinguished not by the likeness of pictures to something else but by some lack of effective differentiation among them. Can it be that -- ironically, iconically -- a ghost of likeness, as nondifferentiation, sneaks back to haunt our distinction between pictures and predicates? (131).
The ghost has some substance. Nondifferentiated pictures are not necessarily "like" each other in the sense that they visually resemble each other, but rather in that they have similar uses; and though this use may not be identified through their likeness to something else, it seems difficult to disentangle from their reference to something else.
It can be said that any formal semantics is based on a structure-mapping. Wittgenstein's so-called "picture theory of meaning" is a prototypical way of presenting the semantics of natural language as a relation between the structure of the linguistic expressions and the (logical) structure of the world. More modern versions of the story use mappings between set-theoretic models or algebraic signatures to achieve a similar result. What is emphasised by Wittgenstein's later work, however, is that there's no definitive, given way of doing the mapping. Various kinds of symbol systems come into being and acquire such mappings only by virtue of being used by communities of people for various, typically communicative ends. Conventions evolve that "standardise" to some extent the ways in which this is done, so that people can usefully generalise their understanding from one case to another, but there is always a good deal of latitude. The organisation of symbols into systems emerges from the development of these conventions, but then it also emerges that symbols and systems have many different kinds of properties at different levels of structural abstraction. Not only that, but there are different ways of structuring the "world" onto which symbol structures are mapped: it can be subjected to different schemes of conceptualisation (what Goodman might think of as different "ways of worldmaking"), some of which may be more conventional than others. Following Gurr (1998) and Gurr et al. (in press) we call these abstract world-representations "a-worlds".
The upshot is that we have a mapping between two structures (a-worlds) that are susceptible of the same general kind of formal description, which allows us to examine particular properties of the mapping. One property that seems to be important has been called systematicity (cf. Gurr et al., op. cit.). A mapping between two structures is systematic, crudely speaking, when the mapping involves and preserves properties and higher-order properties (i.e. properties of properties, such as transitivity etc.) that hold among the entities mapped. Thus a family tree can be based on a systematic mapping in that connections by lines represent parenthood relations (which are both intransitive), whereas being above represents being an ancestor of, which are transitive relations. If lines to represent parenthood were drawn in random directions[4], the diagram would still in principle be usable, but a number of useful topological features of trees would no longer be shared by the diagram, and e.g. ancestorhood would have to be inferred by following multiple parenthood links, rather than being represented directly. Relative to an a-world in which the ancestorhood relation is explicit, this diagram would be less systematic than the tree. Systematicity of this kind is important where diagrams are used for reasoning; but it is also relevant to issues of depiction.
Note here that systematicity is a property of the relation between a-worlds, and not of the abstractions themselves. If both a-worlds are very "flat" and contain only first-order relations, then a mapping that only maps these relations may still be maximally systematic (i.e. isomorphic at all levels). We may feel that a set of parenthood relations just inevitably induces the ancestorhood relation. However, this remains a feature of the domain that we might not have included explicitly in our abstraction; in which case its omission is no fault of a diagram intended to communicate that abstraction. Arguably in such a case the tree, with its tendency to be read as illustrating a transitive relation, would be implying too much.
We said: "especially for analog schemes". Repleteness, as Goodman uses it, seems to apply only to analog schemes, but it can also be considered in relation to notations, such as text. Features like spatial layout seem clearly able to have a function. Petre and Green (1992) discuss the concept of secondary notation. Where there exists a well-defined diagrammatic system, diagrams may often be constructed which go beyond the defined system -- prototypically, items in an electronic chip design may be grouped by experienced designers in ways that indicate useful facts about their relationships. By the standards of the simplest parenthood abstraction, use of the vertical direction to induce ancestorhood in family trees could be seen as a case of secondary notational use of the arrow-based representation. However, it would always be possible to define a new a-world with respect to which the secondary notation is well-defined and hence now "primary". This would also be a system entailing a scheme in which more properties were relevant to symbolic identity, and hence more replete. Though Petre and Green speak of diagrams, the idea of secondary notation appears to cover aspects of text, as in the issue of spatial layout raised above. Of course, natural language is not a well-defined system, so let's consider computer programming languages. These are very commonly defined without regard to the nature of the "white-space" characters between the various lexical items, but whether a character is a space, a tab or a newline has a dramatic effect on the visual appearance of the program code (text), as normally presented. The resulting layout is crucial to the usability of the text for a human reader, precisely because there is a relationship, though it may be intuitive, vague and hard to define, between the layout structure and the abstract structure of the program. This must be in some sense implicit in (derivable from) the unformatted code itself, but in that form it's unavailable to the human user. Layout here implies a secondary representation system with a more replete scheme and a systematic mapping to a more explicit abstraction of the domain structure.
For Goodman, secondary notation may often not be notation. Though a programming language is probably as close to a true notation, in his terms, as anything in practical use will get, the various uses of layout are likely to fail the five criteria[5]. But this is perhaps true of all real notations, including Goodman's favourite example, musical notation. Elkins (op. cit.) discusses a Bach autograph score, suggesting in effect (without of course using this terminology) that many of its features may be seen as secondary notation. Aspects of natural language text, such as layout, the use of various fonts, italics, etc. -- and likewise prosody in speech -- seem plausibly to fall under a similar account. Perhaps also, though this is less clear, the approach will extend to those aspects of language known as "iconicity" among linguists (e.g. Haiman 1985; briefly discussed in Lee and Stenning 1998), where for example the sequencing of items in sentences may relate to temporal ordering, etc. The sharp dichotomy that Goodman sets up between the continuous and the discrete is valuable in theory but often very blurred in practice.
Another view of the tripartite nature of representation is offered by Bull (1994), who combines Goodman's approach with that of Gombrich to produce an interesting emphasis on the notion of a schema, described (in terms that for present purposes are undesirably mentalistic) as "our prior concept of an object's appearance" (214). So we have images, objects and schemata, where the latter form a differentiated symbol scheme which can be used to link images and objects by denoting both. Though taking a very different route, Bull seems to arrive somewhere quite close to Files' position. The schema has very much the role of an interpretant: "We recognise an image correctly if and only if we see it as the schema with which it complies, but the act of recognition does not itself depend on the compliance relationship" (loc. cit.)[7]. What we wish to stress here is that equally the compliance relationship does not depend on the act of recognition. Rather it depends on a structural mapping -- an abstract schema -- that provides for a certain kind of use of the image as a representation. Resemblance and the assistance of visual recognition is just one kind of way in which a mapping can facilitate such use. And this is not to disagree with Elgin (1991) who has noted that
... the scheme/content distinction has come into disrepute, and rightly so. The orders we find are neither entirely of our own making nor entirely forced upon us. There is no saying what aspects of our symbols are matters of conventional stipulation and what are matters of hard fact. For there are few purely conventional stipulations, and no hard facts. (18)
The parallel construction of a-worlds reflects just this kind of mutual interdetermination of our conceptions and our ways of representing them.
Elgin, C.Z. (1991) Sign, symbol and system. Journal of Aesthetic Education 25:1, 11-21.
Elgin, C.Z. (1993) Outstanding problems. Synthese 95:1, 129-140.
Elkins, J. (1993) What really happens in pictures: misreading with Goodman. Word and Image 9:4, 349-362.
Files, C. (1996) Goodman's rejection of resemblance. British Journal of Aesthetics 36:4, 398-412.
Goodman, N. (1969) Languages of Art. Oxford University Press.
Goodman, N. & Elgin, C.Z. (1988) Reconceptions in Philosophy and Other Arts and Sciences. Routledge.
Gurr, C. (1998) On the Isomorphism, or Lack of it, of Representations. In Theories of Visual Languages, K. Marriot and B. Meyer eds. 288--301, Springer-Verlag.
Gurr, C., Lee, J. and Stenning, K. (1998, in press) Theories of diagrammatic reasoning: distinguishing component problems. Minds and Machines.
Haiman, J. (1985) Iconicity in Syntax (ed.). John Benjamin.
Lee, J. (1997) Similarity and Depiction. In Proceedings of the Interdisciplinary Workshop on Similarity and Categorisation, M. Ramscar and U. Hahn (eds.) Dept. of Artificial Intelligence, Univ. of Edinburgh. ISBN 0 907330 27 4.
Lee, J. and Stenning, K. (1998) Anaphora in Multimodal Discourse. In Multimodal Human-Computer Communication, Harry Bunt, Robbert-Jan Beun, Tijn Borghuis eds. 250-263 Springer-Verlag.
Oberlander, J. (1996) Grice for graphics: pragmatic implicature in network diagrams. Information Design Journal 8:2, 163-179.
Petre, M. and Green, T.R.G. (1992) Requirements of graphical notations for professional users: electronics CAD systems as a case study. Le Travail Humain 55, 47-70.
Schier, F. (1986) Deeper into Pictures. Cambridge University Press.
Scholz, O. (1993) When is a picture? Synthese 95:1, 95-106.
DRAFT: Last Updated 24.9.98