Chapter 4: Perception, Cognition, and Affordance

  establishes that complex behavior arises from the interaction between bodies and the environment. Cognition follows action because the mind is not separate from the body, but evolved to compliment it. The body perceives the environment and reacts to it efficiently, based on the most obvious information. 

Information of a Different Sort
Hinton begins by defining cognition as, “The way people understand things … the process by which we acquire knowledge and understanding through thought, experience, and our senses”. This quote shows that Hinton does not see the mind as separate from the body because it reacts to the environment in the same way. There is a coupling between the bodies of simple creatures and their environments, creating a system of complex, advanced behavior. The way we act within the natural world in a series of bodily reactions to information is the same way that we use technology.

The body-environment theory comes from ecological psychology (or Gibsonian psychology): a field “which posits that creatures directly perceive and act in the world by their bodies’ ability to detect information about the structures in the environment”. This information is called physical information, which is what you get when bodies and environments make a system of action and perception. James Gibson defines information as what is in the environment, not what the observer receives, which is perception. We operate with bodies that did not evolve to operate in a digital environment.

A Mainstream View of Cognition
Hinton summarizes the common understanding of cognition: most people think that the brain does most of the thinking and the body is merely the means that connect the brain to the world. This perspective is known as mainstream cognition, disembodied cognition, representationalism, and cognitivism. It models cognition as a series of inputs and outputs, with the brain gathering data through the senses, working with abstract representations, and then translating its instructions to the body.

Mainstream cognition resembles the workings of a computer because cognitive science developed at around the same time as information and computer science. “The computer became not just a metaphor for understanding the brain, but a literal explanation for its function”. As a result, proponents of mainstream cognition think of the brain as sterile instead of organic and connected.

Embodied Cognition: An Alternative View
Embodied cognition is the idea that our bodies are an important part of cognition. Hinton says, “Cognition is truly environment-first, emerging from an active relationship between environment, body, and brain”. In this view, the brain is not the separate holder of knowledge; rather, it is a part of a body that functions necessarily within the world. This view is useful in the user-experience and design fields because it suggests that we should design for bodies, not abstract minds.

Embodied Cognition holds that perception precedes conscious reflection on an experience. Appropriate behavior exists in nature without intelligence, like how a Venus flytrap can act without a brain. The coupling of the organism’s body with the environment makes complex behavior possible. Likewise, humans think better when using the environment, like using sticky notes or sketching. This model holds a continuous loop of action and perception, which drives cognition.

Action and the Perceptual System
In order to understand the environment, we must take action within it. Hinton ties action and perception together, saying, “Context, then, is also a result of action by a perceiving agent, not a separate set of facts somehow insulated from that active perception”. Context is the information we actively perceive. Gibson describes a perceptual system where input and output are a continuous loop. Actions fuel perception—we evolved to constantly change our perspective (action) in order to gain more information (perception). This reaction is automatic. Even in simulated environments, humans will adjust their perspective, even though doing so will not yield new information.

Information Pickup
Gibson uses information pickup “to express how perception picks up, or detects, the information in the environment that our bodies use for taking action”. Using this information, we form assumptions of what our bodies can do within the environment. Just as weather vanes use their structure to adjust to the movement of air, elbow joints evolved for specific environmental actions. This process allows the body to orient itself in the environment based on responses to the environment, not abstract calculation.

Affordance
Affordances are basically the information that we perceive in the environment that tells us what actions we can take in that environment. This information is intrinsic and works with our bodies’ abilities. For example, when viewing a tree branch, we detect affording properties relating to how we can interact with it, like if it would fit well in our hand or if we could use it as a cane. Affordances only make sense within the structure of Gibson’s ecological system because it emphasizes the body.

Hinton explains the properties of affordances. 1) They are also value-neutral, because while some opportunities for action can be good or bad, they are all affordances. 2) Perception of affordance (structure) comes before our ideas about it (category). 3) Affordances exist in the environment even if they are not perceived. 4) Affordances are there even if they are not perceived accurately. 5) Affording information is always in context of other information. 6) Affordances are learned. As we grow up, we learn how to manipulate our own bodies and interact with the world. Because these affordances are not natural, we need to be able to design objects and places that humans can learn how to use.

Directly Perceived versus Indirectly Meaningful
Affordance theory can be applied to help understand all kinds of information. Designers of digital spaces use affordance as a tool to ask how the user interacts with digital environments. The difference between how we perceive physical objects and how we perceive simulated objects can cause confusion on the part of the viewer because we are used to reacting to physical objects, not simulations. We act on the information we get, even if it is a trick or simulation. For a movie, the only affordance is what is produced by the projector—we are allowed the opportunity to view the image, but not to act in the simulated space.

Gibson defines display as “a surface that has been shaped or processed so as to exhibit more information for more than just the surface itself”. Information we get from these artifacts is mediated or indirect instead of direct physical information pickup. This mediated information is not intrinsic and is therefore not considered an affordance. Affordances provide information about what objects are, like how a screen is flat or heavy, but they do not give information on what these surfaces display. Signifiers are also not affordances because they rely on words, not physical structure, to convey information .

Soft Assembly
Affordances create the physical information that cognition acts on. Soft Assembly is when cognition uses different mechanisms to figure out the world: “Various factors of body-environment interaction aggregate on the fly, adding up to behaviors”. We assemble the information we have about the world at the same time as we are gaining more information. Our cognition uses combinations of cognitive loops (“loops of least effort”) using the environment as scaffolding. The mind uses both the body and the world to improve its own abilities. Cognition acts on an assembly principle to work as efficiently as possible. We act on information about the environment, even if it might not be true. This automatic action means that designers need to create environments where the most obvious action is safe.

“Satisficing”
Satisficing describes how we “conserve energy by doing whatever is just enough to meet a threshold of acceptability”. It impacts design because users will act with as little concentration as possible. Not aiming to understand the environment, users will even improvise and use the environment in different ways than intended.

Umwelts
An Umwelt is, “the world perceived and acted upon by a given organism”. Even people with the same senses will live in separate umwelts because everyone has different experiences and interprets information differently. As we create devices that act in response to human variables, those devices also gain umwelts. Hinton mentions that it is important to ask what the device’s umwelts are and how to best translate between human and device umwelts.

Analysis
Hinton’s overall analysis of the relationship between the mind and body is that they form a single unit of perception and reaction with cognition following later. This idea is extremely important to the field of digital writing because understanding how the user thinks should impact the way we design. Humans evolved to reach understanding by acting within the world in a physical, reactionary way. Designs that operate in a similar way to how we interact with the physical world would be much easier to use.

An example of this idea in motion can be seen in Clive Thompson’s article, We’re All Coders Now. In it, he describes how something as abstract as coding can become easy to understand and use if the information is presented in a certain way. He describes his success with the Google App Inventor, which uses pieces of code like Lego bricks to enable users to create apps without knowing code. The app inventor is successful because it takes into account that users think using their bodies. We understand how Legos work and we shuffle ideas around in order to get a better sense of them. Design with an embodied perspective in mind has the potential to create products that are easier to use because they work with how humans evolved to operate.