Multimodality: a Social Semiotic Approach to Contemporary Communication (2010)

IMAG1619 IMAG1620IMAG1621


Societies and their cultures select ‘materials’ – sound, clay, movement (of parts) of the body, surfaces, wood, stone – which seem useful or necessary for meaning-work in the culture to be done


The author begins by explaining that writing, image and colour work together to convey a particular message (mULTIMODALITY), and goes on to introduce Social Semiotics as a theory that deals with meaning in all its appearances (including for example supermarket social standing and assertions of authority). Multimodality has been around for a decade.

Psychology -> gesture

art hist / maths -> image

Multimodality aims to unify, something connected to globalisation – characteristics of one place active and present in another, with awareness of provisionality and instability (eg. MOBILITY). Focus shifts from critique to design. The book examines the benefits/drawbacks of stability, whether to talk about grammar (fixed rules) or ‘resources for representation’. Kress then discusses the reach of this work, concluding that some social, semiotic, communicational principles and dispositions (eg. linguistic) are shared. Translations across modes of representation are difficult so it’d seem to be fairly universal. These seem to be shared:

  1. signs are motivated by conjunctions of form and meaning
  2. …based on the interest of the sign-maker
  3. …using culturally-available resources

Translation between societies of eg. movements and gestures is touched on which might be relevant (p. 11). In this book society emphasises human action in social groups, culture emphasises the effects/products/concepts thereof, work is used in a way that stresses social orientation and aims, changng tools, worker and that which is worked upon. A multi-modal social-semiotic theory is likened to a satellite view of language. The example of a class drawing a cell with nucleus is briefly discussed wrt epistemological commitment.

2, the social environment of contemporary communication – the author sets out certain aims for a social-semiotic theory of communication, for autonomy and awarenes. The relations of state and market had certain effects on conceptions of the individual as ‘citizen’ and ‘consumer’, which theories of communication must have a stance in relation to. Historical censorship by power is reduced by eg. the internet, which will likely be debated. The debate has been framed in terms of class vs. lifestyle – Kress favours ‘life-world’. Power in domains of communication have tended to shft from vertical to horizontal. Authorship in a cut-and-paste age is under debate. The contemporary media landscape is marked by:

  • affordances of participation
  • global AND local reach
  • user-created content
  • accessibility/connectivity/mobility/ubiquity
  • convergence of representation, production, communication in technologies/devices
  • multimodality

User-generated content signals a shift from consumption or critique to design. differing patterns of participation and production will create different engagements and agency for different social groups. The landscape is characterised by

  1. forms of knowledge production (wiki, web 2.0)
  2. forms & principles of text-making composition eg. cut n paste
  3. social & semiotic blurring: the dissolution, abolition, disappearance of frames and boundaries: epistemological/ontologicak -> reality tv and focus on representational conventions over events, social-interactional -> genre boundaries erased, power/authority/convention -> convention now can’t ensure adherence to representational conventions, knowledge -> more complex relationship with information, unstable life-worlds meaning things are less predictable. Knowledge is now a tool made from info. All this means we need navigational aids for discrimination and text-making.

Every event of communication is unpredictable, so the approach must be rhetorical: more than just competent in performance and offering possibility of critique, making an assessment of the message-maker’s interest, audience’s characteristics, semiotic requirements of issue nd resources available, best means of dissemination. Design matches interests to resources. Production implements it, with semiotic (form-as-content), conceptual (content-as-concepts) and affective (semiosis-as-expressive of interest) features simultaneously. A rhetorical approach is based on agency of maker/remaker of messages, who shapes knowledge. Sites of appearance/dissemination are associated with particular characteristics of power and agency, processes of engagement, what is engaged with etc.

Multimodality is concerned with the aptness of a means of representation and how they work together. Style is the politics of choice. Aesthetics is the politics of style. Ethics is the politics of value/evaluation. Mobility/portability are briefly discussed wrt surveillance and need for choice. Pace vs. humanity.

Metaphors provide guides/framings for thinking.All signs are newlymade metaphors for particular audiences.

3. Communication: shaping the domain of meaning. Communication is semiotic work, ie changes worker, tools etc. and a prompt (touch, gaze, word) is responded to via interest through attention -> framing -> interpretation. The decoder/encoder model of communication was challenged by Barthes’ Death of the Author (1968). Kress says that in communication, members of a community participate in the renewing, remaking and transformation of their social environment from the perspective of meaning. The social is constantly articulated in (material) semiotic form, recalibration closing gaps quickly between social structures and semiotic accounts thereof to give the illusion of continuity, but producing slightly new accomodations every time via divergences between groups/individuals.

The social is the motor for communicational/semiotic change, constant remaking of cultural/semiotic resources, and production of the new.

Communication is always a response by one participant to a prompt by other participants in social events.

Communication is the response to a prompt; and communication only happens when there is ‘interpretation’ (see Kress and van Leeuwen 2001)

Hm, how does that work with presentations? Good q.

Sthg only becomes a prompt through interpretation, but refusal to engage is communication, too. Communication is two-stage – there’s the maker bit and the interpreter bit

sign-complex -> message intended as prompt ->attention/engagement -> response

message -> interest -> attention -> engagement -> selection -> framing -> transofrmation -> new inner sign.

Historically, reading can be comfortably fit within the Saussurean Sender-> Message -> Receiver schema. Kress discusses a page from CBBC website, which allows the viewer to do some of their own semiotic work of design. Generational differences are identified and reading in sites of power/institutions is noted. ‘Maps’ drawn of an exhibition (complex sign) show differences in attention.

This theory diminishes neither the significance of the semiotic work of the maker of the initial message, the rhetor, nor that of the interpreter: the work of design that fashions the ‘ground’ on which interpretation takes place is one essential element in the two-part structure of the process of communication.

Museums are sites for education and entertainment.

One sign-complex provides the ground, as message. The other sign complex via attention, selection, framing etc. then transduction and transformation -> new, inwardly focusing sign complex. Either could be considered the rhetor, but Kress settles on calling the first rhetor and second interpreter. Rhetoric is the politics of communication and politics is the attempt to shape and regulate social relations by means of power (a lot of this book seems to be explained backwards. Why put the heavy concept first and the sensible explanation second? I’ve seen that a few times, now, and it seems almost disingenuous). The grooves of convention have been worn away; political and social instability means things are provisional. There’s a look at the increase of image in teaching materials, diagrams etc, where they appear most. Potentially v useful.

Representation is about my interest in the world and my desire to give material realisation to it; communication is about making that available to others. Rhetoric and design (as rhetoric’s implementation) do this. There’s some nice stuff about lighting at exhibitions. Representation remakes the resources for making meaning, and shapes those who make it and is, apparently, ontologically and epistemically problematic (for power?). Communication reconstructs the social environment, changes the potentials of action, agency etc and is politically problematic (for power?). ‘Normal times’ are when conventions are known and comfortably adhered to. Question – is there a need to differentiate presentation from representation? is everything a re-presentation of other signs?

There is the possibility of seeing sign-making either as constantly transformative, as a constant remaking – the position which I adopt; or we assume that sign-making, at times, proceeds from a ‘freash start’. p. 53

4. A social-semiotic theory of multimodality: from a linguistic to a multimodal socia-semiotic theory of meaning and communication. Multimodality names both a field of work and a domain to be theorised. In semiosis, signs are made (socially) rather than used (this distinguishes social-semiotic theory from other forms of semiotics). Social Semiotics derives from Michael Halliday (1978, 1984) and strands thereof take a linguistic or a semiotic approach. Kress/van Leeuwen approach assumes:

  • signs are newly made
  • signs are motivated, not arbitrary relations of meaning and form – the choice is made via aptness/best fit
  • relation of form and meaning is based on/arises from the interest of makers of signs
  • the forms/signifiers thus made in social interaction become part of the semiotic resources of a culture

A couple of children’s drawings are discussed and there’s reference here to Lakoff/Johnson (1982), though this (Kress) approach is more social and less cognitivist.

Kress outlines the difference between a linguistic, pragmatic/sociolinguistic and a social-semiotic approach to the question “I wanted to ask, could I have an extension on my essay?”. The motivation for certain elements is said to be the use of a form that mirrors/parallels the ‘meaning shape’ of what is meant – temporal distance for social distance, etc., an iconic relation. He summarises: linguistics -> forms and their relations, pragmatics -> social circumstances/participants/environments/effects, social semiotics -> interest/agency/social meaning-making/signs-as-metaphors/meaning potentials of semiotic forms.

Something of the ideology of the theory is discussed – social use, all media considered. Then we have a look at Peirce (interpretant) and Saussure (outer-inner) (p.62). Kress explores Wittgenstein’s chess piece button example, rejects with arbitrariness, disgrees with its part (symbol) in Peirce (that might be a bit much, surely it can be allowed to play a part, he has it in conjunction with other things…?). Re Saussure:

the signifier of ‘tree-ness’ is not a sequence of sounds, that is, not [a:br], but an existing lexical-item-as-signifier ‘tree’ used in its potential for becoming a new sign. The meaning-potential of the signifier ‘tree’ is the sum of all the instances in which I have encountered the sign ‘tree’ as signifier: that enables me to make a prediction about its aptness as a signifier for the new sign that I want to make now. The signified TREE and the signifier ‘tree’ are elements at the same level and of the same kind: not as in Saussure’s assumption where one is a semantic entity and the other is a phonetic one, one an entity of meaning and the other an entity of sound.

Arbitrariness is replaced by motivation… why is it chosen…

A child’s story and drawing of a certain event take different forms, one sequential, one spatial, according to the affordances/logics of each form. A bit about imitating gestures, p.76

In the engagement with any sign, the materiality of modes – where sign and mode are understood broadly – interacts with the physiology of bodies

5. MODE! Different modes have different potentials – eg. writing has graphic resources like how it’s written, and syntactic, textual social-semiotic resources. The different potentials of speech and writing are discussed. (jesus, this book wasn’t written when I was learning about the medium specificity hypothesis) Useful stuff on p82 about what things can do.

Societies and their cultures select ‘materials’ – sound, clay, movementm(of parts) of the body, surfaces, wood, stone – which seem useful or necessary for meaning-work in the culture to be done.

Modes are socially shaped, with selections from these potentials. What can constitute a mode is determined socially, by the users, and formally, by the theorisers. Kress considers whether layout can be considered a mode, which is an interesting question. There’s a great image of ‘writing’ by small children showing how they learn how writing works – that it fits between the lines, is repetitive, linear, progressive, etc. There could be a Foucauldian power thing in here too.

2016-01-23 16.24.16

6. Meaning as resource: ‘naming’. Meaning-resources: discourse, genre and mode.

The basic idea of multi-modality is interesting and relevant but right now it’s proving a little hard to pin down this particular text enough for it to be useful – people have talked about the affordances of different media before, surely..? The distinction between mode and medium might perhaps be important here.



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s