John Lee
Human Communication Research Centre
and EdCAAD, Dept. of Architecture
University of Edinburgh
2 Buccleuch Place
Edinburgh EH8 9LW
Scotland, UK
The distinction between words and pictures is approached via Nelson Goodman's theories about symbol systems and notations, denotation and exemplification. It is argued that his attempt to draw a purely syntactic distinction fails. An attempt is made to reconcile Goodman with a notion of pictures as based on interest-relative structure-mappings. Comparisons are drawn between e.g. Goodman's concept of "repleteness" and the "systematicity" of structural mappings.
The locus classicus of comparative study between graphical and linguistic systems is Nelson Goodman's Languages of Art [2]. Goodman is concerned with a general issue about how representation works--how marks on paper are related to various kinds of things in the world[1]--in a range of cases such as pictures, music and other kinds of notation. His cornerstone is to establish what distinguishes a "notational symbol system" from other kinds of symbol system. His approach forms the prototype for most later formal theories in this area, in as much as he considers even pictures to be symbol systems which represent not in virtue of any notion such as resemblance, but due to their being subject to certain systematic rules of use.
According to Goodman, there are five basic conditions required for a symbol system to be notational. The first two of these are syntactic, the others semantic.
1. It must consist of symbols (utterances, inscriptions, marks) which form equivalence classes (characters) on the basis that they can be exchanged without syntactical effect. Alphabets are a prototypical example--any "a" is as good as any other; they are "character-indifferent", and the characters have to be disjoint, so that no mark qualifies as an instance of more than one character. In general, Goodman takes compound inscriptions (e.g. sentences) to be characters as well.
2. Characters have to be "finitely differentiable" (or "articulate") in the sense that their disjointness is feasibly testable, which rules out, in particular, "dense" systems where any two (ordered) characters have another between them.
3. Notational systems must be unambiguous, so that the extension (which Goodman calls the "compliance-class") of an inscription is invariant with respect to time, context, etc.
4. The compliance-classes of all characters must be disjoint. (Also, the system will ideally be non-redundant.)
5. Compliance-classes must also be finitely differentiable. Thus, for example, any system which is "semantically dense", in that its compliants form an ordering such that any two have another between them, is excluded.
Goodman elaborates these points in relation to clocks and pressure gauges, which measure quantities that are infinitely variable. Here, the semantic domain can always be seen as dense, and if there are no marks on the dial, then there is no syntactic differentiation of characters, so the representation system is clearly non-notational. It can become syntactically notational if, say, dots are distributed around the dial and each is taken to be the centre of a disjoint region such that the pointer appearing anywhere within that region counts as an inscription of a certain character. If the ranges of pressure correlated with these regions are also disjoint (and articulate), then the system meets the semantic requirements as well, and hence is simply a notation. On a clock face, the hour hand is typically used notationally in this way, whereas the minute hand may be seen as marking the absolute elapsed time since the passing of a particular mark, and hence is non-notational.
Diagrams, one might think, are typically non-notational. Goodman observes that many topological diagrams are in fact entirely notational. This also applies e.g. to many drawings used in architecture and design, where although there may be a non-notational impression of form, measurements etc. are always given and the use of the drawing becomes largely notational. Road maps are a common example of mixed diagrams, with both notational and non-notational aspects. Non-notational diagrams are equivalent to two-dimensional models, taking the latter term (which in general can mean "almost anything from a naked blonde to a quadratic equation") to exclude descriptions and samples. Models, like diagrams, of molecules are usually entirely notational; others range all the way to being entirely non-notational.
Goodman approaches the difference between diagrams and pictures by introducing a further notion of "repleteness". A symbol is relatively replete if a relatively large number of its properties are involved in its identity as a symbol; something is more a picture, and less a mere diagram, if there is less about it that can be changed without making it into a different picture. This concept receives more detailed discussion below.
Goodman's general view is summarised as follows:
Descriptions are distinguished from depictions not through being more arbitrary but through belonging to articulate rather than to dense schemes; and words are more conventional than pictures only if convention is construed in terms of differentiation rather than of artificiality. (230-231)
According to his own account, however, Goodman is not here trying to define the pictorial. Writing much later, in "Representation re-presented" ([3], ch. VIII), he says:
Nowhere in my writing to date have I proposed a definition of depiction, but have only suggested that the everyday classification of symbols into pictures and nonpictures is related in an important way to the line between symbols in a dense or 'analog' system and those in a finitely differentiated or 'digital' system. [3] (123)
This characterisation is then sharpened up somewhat by noting that the distinction between analog and digital does not depend on the semantics of the system. Considering only the syntactic aspect (called a scheme, where clearly a scheme, being susceptible of having different denotations assigned to it, can belong to more than one system), Goodman notes that digital and analog schemes can be categorised on the basis of differentiation among the symbols in the scheme. Goodman is thus led to claim that the pictorial can be distinguished from the verbal on a purely syntactic basis, despite the apparently paradoxical facts that "all symbols belong to many digital and analog schemes", and "some schemes consisting entirely of pictures ... are digital" [3] (130). The key to resolving this paradox is said to lie in considering the comprehensive or full scheme for a whole language (e.g. English) or pictorial system (e.g. our pretheoretical idea of pictures).
This vehemently expounded argument begins to call into question the very existence, or at least definiteness, of the system of rules; the syntax and semantics. The identification--and hence identity--of a word, or its location in a grammatical category, becomes open to question. If we look back at Goodman's approach to defining a syntax, we note that it depends on discriminable marks that fall into equivalence classes and are interpreted unambiguously. In fact, few symbol systems in practical use will meet these criteria, and the observations in the previous paragraph serve to emphasise that even when they may appear to this is likely to be an illusion. How, in fact, are the relevant equivalence classes identified?--By the patterns of use that the symbols are subject to, e.g. what can be exchanged "without syntactical effect". But such effects can only be identified on the basis of a certain amount of theorising, which in generating the distinction between syntax and semantics (and that which is neither) departs from the reality of practice where context and relation to experience are everything. Any distinction so generated is surely to be regarded as bounded and perhaps temporary, certainly subject to revision in the face of different kinds of usage.
In these circumstances, can we really speak of a comprehensive symbol scheme? Difficult as this must be for the symbols of a language, it seems still more so for those constituting a pictorial system. As Goodman himself emphasises, one and the same picture may appear in one situation as a digital character, in another as an analog picture. It seems manifestly implausible that we can tell which is which on purely syntactic grounds, because this requires us to establish when the picture can be substituted by another; and even if this can be found out from an agnostic scrutiny of patterns of usage, it surely still depends on what the picture is taken to represent. On the one hand, it is deeply problematic to identify the system that is at hand when any symbol is being considered; on the other hand, as far as pictures are concerned it appears that when used analogically each is a unique exemplar of a symbol and hence, as Elkins observes [4], that "there is very little sense in calling non-notational images 'systems'" (361).
A defence of the syntactic approach is mounted by Scholz [5] (101-2) on the basis that pictures are common enough which do not denote at all--e.g. pictures of fictional objects. We can accept this without finding it very helpful. In all symbol systems there's a sense in which what something means is distinct from the question of whether anything corresponds to this. Elgin [6] (135), responding to Scholz, makes a related point in observing that reference, as understood by herself and Goodman, encompasses more than denotation, including e.g. exemplification, expression and allusion. For these or other reasons, we surely have to insist that symbols which fail to denote "real world" objects are not thereby shown to lack interesting semantic properties; but also it is hard to see that syntactic properties alone can suffice to distinguish pictures from other symbols.
Goodman worries that
The pictorial is distinguished not by the likeness of pictures to something else but by some lack of effective differentiation among them. Can it be that--ironically, iconically--a ghost of likeness, as nondifferentiation, sneaks back to haunt our distinction between pictures and predicates? [3] (131)
The ghost has some substance. Nondifferentiated pictures are not necessarily "like" each other in the sense that they visually resemble each other, but rather in that they have similar uses; and though this use may not be identified through their likeness to something else, it seems difficult to disentangle from their reference to something else.
It can be said that any formal semantics is based on a structure-mapping. Wittgenstein's so-called "picture theory of meaning" is a prototypical way of presenting the semantics of natural language as a relation between the structure of the linguistic expressions and the (logical) structure of the world. More modern versions of the story use mappings between set-theoretic models or algebraic signatures to achieve a similar result. What is emphasised by Wittgenstein's later work, however, is that there's no definitive, given way of doing the mapping. Various kinds of symbol systems come into being and acquire such mappings only in virtue of being used by communities of people for various, typically communicative ends. Conventions evolve that "standardise" to some extent the ways in which this is done, so that people can usefully generalise their understanding from one case to another, but there is always a good deal of latitude. The organisation of symbols into systems emerges from the development of these conventions, but then it also emerges that symbols and systems have many different kinds of properties at different levels of structural abstraction. Not only that, but there are different ways of structuring the "world" onto which symbol structures are mapped: it can be subjected to different schemes of conceptualisation, some of which may be more conventional than others. Following Gurr [8] [9] we call these abstract scheme- and world-representations "a-worlds".
The upshot is that we have a mapping between two structures (a-worlds) that are susceptible of the same general kind of formal description. The mapping constitutes denotation, going from the abstraction of the representing scheme (e.g. some formalisation of a type of graphics) to an abstraction of the represented domain. The formalisation allows us to examine particular properties of the mapping. One property that seems to be important has been called systematicity (cf. [9]). A mapping between two structures is systematic, crudely speaking, when the mapping involves and preserves properties and higher-order properties (i.e. properties of properties, such as transitivity etc.) that hold among the entities mapped. Thus a family tree can be based on a systematic mapping in that connections by lines (intransitive) represent parenthood relations (intransitive), whereas being above represents being an ancestor of, which are transitive relations. If lines to represent parenthood were drawn in random directions[3], the diagram would still in principle be usable, but a number of useful topological features of trees would no longer be shared by the diagram, and e.g. ancestorhood would have to be inferred by following multiple parenthood links, rather than being represented directly. Relative to an a-world in which the ancestorhood relation is explicit, this diagram would be less systematic than the tree. Systematicity of this kind is important when using diagrams for reasoning; but it is also relevant to depiction.
Note here that systematicity is a property of the relation between a-worlds, and not of the abstractions themselves. If both a-worlds are very "flat" and contain only first-order relations, then a mapping that only maps these relations may still be maximally systematic (i.e. isomorphic at all levels). We may feel that a set of parenthood relations just inevitably induces the ancestorhood relation. However, this remains a feature of the domain that we might not have included explicitly in our abstraction; in which case its omission is no fault of a diagram intended to communicate that abstraction. Arguably in such a case the tree, with its tendency to be read as illustrating a transitive relation, would be implying too much.
We said: "especially for analog schemes". Repleteness, as Goodman uses it, seems to apply only to analog schemes, but it can also be considered in relation to notations, such as text. Features like spatial layout seem clearly able to have a function. Petre and Green [10] discuss the concept of secondary notation. Where there exists a well-defined diagrammatic system, diagrams may often be constructed which go beyond the defined system--prototypically, items in an electronic chip design may be grouped by experienced designers in ways that indicate useful facts about their relationships even though these groupings are formally undefined. By the standards of the simplest parenthood abstraction, use of the vertical direction to induce ancestorhood in the family trees discussed in the last section could be seen as a case of secondary notational use of the arrow-based representation. However, it would always be possible to define a new a-world with respect to which the secondary notation is well-defined and hence now "primary". This would also be a system entailing a scheme in which more properties were relevant to symbolic identity, and hence more replete. Though Petre and Green speak of diagrams, the idea of secondary notation appears to cover aspects of text, as in the issue of spatial layout. Since natural language is not a well-defined system, let's consider as an example computer programming languages. These are very commonly defined without regard to the nature of the "white-space" characters between the various lexical items, but whether a character is a space, a tab or a newline has a dramatic effect on the visual appearance of the program code (text), as normally presented. The resulting layout is crucial to the usability of the text for a human reader, precisely because there is a relationship, though it may be intuitive, vague and hard to define, between the layout structure and the abstract structure of the program. This may be in some sense implicit in (derivable from) the unformatted code itself, but in that form it's unavailable to the human user. Layout here implies a secondary representation system with a more replete scheme and a systematic mapping to a more explicit abstraction of the domain structure.
For Goodman, secondary notation may often not be notation. Though a programming language is probably as close to a true notation, in his terms, as anything in practical use will get, the various uses of layout are likely to fail the five criteria[4]. But this is perhaps true of all real notations, including Goodman's favourite example, musical notation. Elkins [4] discusses a Bach autograph score, suggesting in effect (without of course using this terminology) that many of its features--the ways notes are grouped, etc.--may be seen as a more replete secondary notation. Aspects of natural language text, such as layout, the use of various fonts, italics, etc.--and likewise prosody in speech--seem plausibly to fall under a similar account. Perhaps also, though this is less clear, the approach will extend to those aspects of language known as "iconicity" among linguists (see e.g. [11]; briefly discussed in [12]), where for example the sequencing of items in sentences may relate to temporal ordering, etc. The sharp dichotomy that Goodman sets up between the continuous and the discrete is valuable in theory but often as blurred in practice as even the sharp formal edges of well-defined symbol systems.
Another view of the tripartite nature of representation is offered by Bull [14], who combines Goodman's approach with that of Gombrich to produce an interesting emphasis on the notion of a schema, described (in terms that for present purposes are undesirably mentalistic) as "our prior concept of an object's appearance" (214). So we have images, objects and schemata, where the latter form a differentiated symbol scheme which can be used to link images and objects by denoting both. Though taking a very different route, Bull seems to arrive somewhere quite close to Files' position. The schema has very much the role of an interpretant: "We recognise an image correctly if and only if we see it as the schema with which it complies, but the act of recognition does not itself depend on the compliance relationship" (214)[6]. We wish to stress here that equally the compliance relationship does not depend on the act of recognition. Rather it depends on a structural mapping--an abstract schema--that provides for a certain kind of use of the image as a representation. Resemblance and the assistance of visual recognition is just one kind of way in which a mapping can facilitate such use. And this is not to disagree with Elgin [15], who notes that
... the scheme/content distinction has come into disrepute, and rightly so. The orders we find are neither entirely of our own making nor entirely forced upon us. There is no saying what aspects of our symbols are matters of conventional stipulation and what are matters of hard fact. For there are few purely conventional stipulations, and no hard facts. [15] (18)
The parallel construction of a-worlds reflects just this kind of mutual interdetermination of our conceptions and our ways of representing them.
Exemplification is in no way limited to linguistic labels. A diagram has some given denotation; it is then exemplified by its referent(s). The family-tree diagram is exemplified by the set of relationships in the depicted family. This is again dependent on the particular abstractions that are invoked on either side of the mapping: the relationship of being father of will exemplify the spatial relationship of being above only where the latter has been established as denoting the former in some symbolic system. Systematicity is therefore as relevant to exemplification as to denotation. In a fully systematic mapping between two sets of abstractions--an isomorphism--exemplification is the exact converse of denotation. Lapses in systematicity raise dangers of misunderstanding in both directions.
Goodman notes that the taylor's swatch exemplifies only certain properties of the bolt from which it comes, such as the colour and weave, and not e.g. being made on a Tuesday. This seems not unlike the doctrine of constitutive and contingent properties: here, the day of manufacture is contingent with respect to exemplification, which is as much as to say that no such label as "made on a Tuesday" is part of the abstract description (of both the swatch and the bolt of cloth) which is in use for present purposes. We assume that there is an abstract label--describing, say, the weave--which refers to some property of both the swatch and the bolt, and this label is then exemplified by both of its referents. We now see that the role of the label here is similar to that of the schema discussed in the last section, denoting both the referring symbol and the thing referred to. The swatch may loosely be said to exemplify the bolt at best by some sort of analogy, but it is the possibility of some such connection that supports our normal talk of swatches as samples of bolts.
A suggestion one might make then, along the lines of Bull's use of schemata, is that pictures and their objects be treated as related via common referenthood with respect to some abstract set of labels. We would then say that a picture depicts what it does because we can describe both the same way: a picture and its object would both exemplify the same description. In a sense, Goodman does say this, but avoids the extra layer of abstraction by maintaining that a picture can be a non-linguistic label that denotes, and hence exemplifies, itself as well as its object ([2], 59ff; see also Elgin [16], 77-8). This situation is uncommon with words: "sesquipedalian", being a word that means[7] "a long and ponderous word", denotes and exemplifies itself, as does "polysyllabic", but relatively few words behave thus. Perhaps all pictures do? Elgin seems to imply as much: "In exemplifying, a symbol in [a pictorial] system functions as a label that denotes itself and the other things that match it", and again "[t]wo symbols exemplify the same label if they match each other and refer to the same shared feature" [16] (78). She also applies this idea to rhythms, musical phrases etc. It may now appear that the essential arbitrariness of denotation has been usurped, though something of this seems natural in cases of self-reference[8], and also that the notion of "matching" is suspiciously like resemblance, which with Bull we agreed should be independent of compliance (and hence exemplification). But again an alternative is structure-mapping at some appropriate level of abstraction. If pictures and other such structures are somehow necessarily self-referential, this marks them out from words in a rather interesting way, and certainly in a way consistent with the idea that their reference is based on structure-mapping, since of course anything structured shares its own structure. We have almost the appearance of Goodman (and Elgin) offering, without explicit mention of structure, a nonetheless structure-based account; and one, moreover, in which the structures that matter are just those that serve the interests of the users of the symbolic system that they and their uses determine.
I look up and see on the wall a painting by Cezanne which appears to depict a group of women bathers. It is important to my understanding and appreciation of the work that I see it as a picture of such a group, but it does not matter whether there ever actually existed such a group[10], or whether if so they were very much as depicted. With respect to groups of women, the nature of this painting can be compared to that of a somewhat abstract diagram, and perhaps one way to think of this is that it exemplifies a group of women. It exemplifies the label "group of women", which due to the self-referentiality of pictures also gives it the denotational role of that label. The seeming sophism here can perhaps be dissolved by considerations of structure. Properly to exemplify the label "group of women", one might think, something should actually be a group of women, so what the picture really exemplifies is the label "group-of-women-label"; but now if we accept that (at least for pictures) to exemplify is to share structure at a suitable level, it becomes possible to collapse this threatening regress.
Here, systematicity and repleteness seem again to come apart. This picture has very many properties--line, colour, composition, etc.--that are critical to its appreciation but are of no significant representational interest. In as much as these properties are constitutive of the identity of the painting as an artwork, but largely contingent in relation to what it might depict or exemplify, we see how thoroughly repleteness is a relative notion: the painting is replete or not only as considered for the time being as a particular kind of symbol in a particular scheme. For a fuller account of its aesthetic qualities we will have to look beyond its symbolic aspects. Here, however, we restate: notwithstanding that the precise semantics is in many respects unimportant, the representational nature of the work in so far as it is considered to be a symbol is central. The relevant scheme (syntax) cannot be coherently identified except as part of some particular system (including semantics), and once again the system will ideally exhibit thoroughgoing systematicity.
2. Goodman, N. (1969) Languages of Art. Oxford University Press.
3. Goodman, N. & Elgin, C.Z. (1988) Reconceptions in Philosophy and Other Arts and Sciences. Routledge.
4. Elkins, J. (1993) What really happens in pictures: misreading with Goodman. Word and Image 9:4, 349-362.
5. Scholz, O. (1993) When is a picture? Synthese 95:1, 95-106.
6. Elgin, C.Z. (1993) Outstanding problems. Synthese 95:1, 129-140.
7. Schier, F. (1986) Deeper into Pictures. Cambridge University Press.
8. Gurr, C. (1998) On the Isomorphism, or Lack of it, of Representations. In Theories of Visual Languages, K. Marriot and B. Meyer eds. 288-301, Springer-Verlag.
9. Gurr, C., Lee, J. and Stenning, K. (1998, in press) Theories of diagrammatic reasoning: distinguishing component problems. Minds and Machines.
10. Petre, M. and Green, T.R.G. (1992) Requirements of graphical notations for professional users: electronics CAD systems as a case study. Le Travail Humain 55, 47-70.
11. Haiman, J. (1985) Iconicity in Syntax (ed.). John Benjamin.
12. Lee, J. and Stenning, K. (1998) Anaphora in Multimodal Discourse. In Multimodal Human-Computer Communication, Harry Bunt, Robbert-Jan Beun, Tijn Borghuis eds. 250-263 Springer-Verlag.
13. Files, C. (1996) Goodman's rejection of resemblance. British Journal of Aesthetics 36:4, 398-412.
14. Bull, M. (1994) Scheming schemata. British Journal of Aesthetics 34:3, 207-217.
15. Elgin, C.Z. (1991) Sign, symbol and system. Journal of Aesthetic Education 25:1, 11-21.
16. Elgin, C.Z. (1983) With reference to reference. Hackett Publishing Co.
17. Oberlander, J. (1996) Grice for graphics: pragmatic implicature in network diagrams. Information Design Journal 8:2, 163-179.