01
They Did Not Expect Him
In 1884 Ilya Repin showed the public a painting in which absolutely nothing happens: a room, light from a window, a laid table in the background, worn wallpaper, and in the doorway a thin man in borrowed clothes who has just come in from the street. A woman in an armchair has begun to rise towards him and has frozen halfway through the movement; the boy at the table has already recognised the newcomer, the girl beside him has not yet, and the whole canvas is built on those two seconds between recognition and non-recognition, stretched out by Repin forever.
It is called They Did Not Expect Him. Who has returned, from where and how long he was gone, the viewer is not told at all, and the face of the main figure the artist kept repainting for several more years, as if he himself had not quite decided whom he had let into that room.
For eighty years the painting hung as a painting. In the sixties it was turned into a measuring instrument, and what it measured was far from art.
Alfred Yarbus worked at the Institute for Information Transmission Problems in Moscow and practised a craft whose description would make a modern person wince: he recorded where a living eye was looking, with a precision nobody before him had achieved.
The apparatus matched the task. Under local anaesthetic a suction cap with a tiny mirror was placed on the eyeball, a beam of light bounced off it onto moving photographic film, and every turn of the eye drew a line on it, while the subject sat with a bar clamped between his teeth so as not to turn his head, and looked at whatever he was shown. Today the same thing is recorded with an infrared camera from half a metre away, and with lower precision; back then precision was paid for with the volunteer’s eye.
Before any painting, Yarbus used this suction cap to discover something that still seems to me the strangest thing in the physiology of vision. If an image is fixed so that it travels with the eye and falls all the time on the same retinal cells, the picture fades out completely within a few seconds, and the person is left sitting with open eyes in front of a blank grey field; that is, vision is built so that for it the motionless is equivalent to the absent: it runs on change alone, and everything that has frozen it crosses out as unnecessary.
The eye passes the brain the difference and writes down to zero everything that has stopped changing. Remember this habit: we will keep running into it throughout the book, on every storey, all the way up to quantum mechanics.
Next Yarbus took Repin’s canvas and set up an experiment that opticians usually never get round to.
One and the same person was shown one and the same painting seven times in a row, for three minutes each time, and seven times the trajectory of his gaze was written onto film; a single detail changed each time: the sentence spoken to the subject before the showing.
The first time he was simply invited to look. Then he was asked to estimate how well off this family was and to determine the age of each person present; then to guess what they had been doing before the visitor arrived, to memorise what everyone was wearing, to keep in his head where everything stood in the room, and finally to estimate how long the newcomer had been away.
Seven questions, one painting, one observer.
The film gave back seven different drawings.
Seriously different: seven dissimilar patterns in which, in essence, only the faces coincided (a person looks at faces whatever the task), while everything else was redrawn anew for each question asked. Ask about money, and the gaze went wandering over the furniture, the walls, the clothes; ask about age, and it gathered on the faces and hardly left them; ask where things stood, and it set off round the room along the perimeter, the way a person does who will later have to arrange the furniture here from memory.
Three minutes is quite a long time if the only thing required of you is to look at one painting: the eye manages several hundred fixations in that time. And out of these several hundred fixations, which could have been spread over the canvas in any way at all, the subject each time assembled a new combination, tailored to the sentence he had heard a second earlier.
This was published in 1965 in the book The Role of Eye Movements in the Process of Vision, came out in English two years later, and since then Yarbus’s drawings have been reprinted, it seems, by everyone who has ever had to give a lecture on attention.
The usual conclusion drawn from this concerns art: one must know how to look, they say. For such a film this conclusion is too small.
What the film records is this. The same canvas hung all the time, the same light fell, the same eye worked, and the three minutes were the same; the only thing that changed was a few words spoken aloud before the person was shown the painting, and following those words everything he ended up seeing moved somewhere else.
The picture that finally appeared before him had been assembled to order, and the order arrived before the image.
The arithmetic here is not cheerful. In three minutes the eye touches with high resolution a negligible fraction of the canvas: there are several hundred fixations, each covering a patch the width of a palm if you stand at the usual museum distance, and all the space between fixations slides across the retina at a quality at which faces can no longer be made out and captions cannot be read. Yet the subject is completely convinced that he has seen the whole painting, and if you ask him he will describe to you the wallpaper, the table, the faces and the light from the window. He is not even thinking of lying; it is simply that what was missing was filled in for him, and filled in solidly, from expectations, memory and general ideas of what rooms look like.
Perception is usually understood as reception: the outside world delivers, the senses take, the brain deals with what has been taken. Yarbus’s film records the reverse sequence: a person sends a request, and the world answers that request, while everything that was not asked about arrives as a cheap fake that is thoughtfully passed off to you as the original.
This story, however, has been considerably knocked about, and that should be told right here, before it starts to sound too neat.
In Yarbus, the question determined the trajectory, and the reverse suggested itself: take someone else’s trajectory, look at it, and you will name the question the person was asked. For half a century this was taken for granted, and for just as long nobody bothered to check.
It was checked in 2012. Michelle Greene, Tommy Liu and Jeremy Wolfe showed people photographs for ten seconds, gave them one of four tasks, recorded their eye movements, and then set a classifier loose on the recordings; it determined perfectly well which photograph the person was looking at, almost as well recognised which subject it was dealing with, and failed completely at the task, staying at the level of random guessing. They raised the viewing time to a minute and got the same. Then they gave the trajectories to living people and asked them to guess the task, and the living people did no better than the machine. The paper came out in Vision Research under the title it deserved: “Reconsidering Yarbus: A failure to predict observers’ task from eye movement patterns.”
Two years later it turned out that the burial had been premature: Ali Borji and Laurent Itti took the same data, changed the processing method and extracted the task from it, titling their paper “Defending Yarbus,” and in parallel another group went the same way and got a similar result.
This squabble is worth sorting out because it separates what is solid from what holds only on someone’s word.
What is solid is that the question reshapes the trajectory of the gaze: Yarbus measured it, others have reproduced it many times, and none of those arguing called it into question.
What turned out to be shaky is the reverse reading, where one tries to reconstruct the question itself from the trajectory: the signal is weak and depends on the processing method, on the viewing time and on how different the tasks were in the first place. A detail also surfaced that explains half the argument. Yarbus had three minutes and a painting with a plot that demands scrutiny, while the refuting experiment had ten seconds and ordinary photographs; in ten seconds the eye makes two or three dozen fixations, and any two strategies manage in that time to do roughly the same thing, going round what is most conspicuous.
The outcome, to be honest, is twofold. Observation works as a request: verified and standing firm. The request can be recovered from the result of the observation: an open question, and the answer depends on how much time you gave the observer and how different the things you asked him were.
All of the above happened to a person with a suction cap on his eye. But the condition that gave rise to this story has nothing whatever to do with biology: Yarbus’s subject chose where to look solely because he could not look at everything in detail at once. The field is large, the means are finite, so selection is needed, and every selection needs a rule by which it is made. The rule turned out to be the question.
Now look at what this construction is made of: finite means, excessive input and the need to decide which part of the input to examine in detail. Nothing living, human or visual remains in this set at all, which means that every system in which the same three conditions come together will end up in the same position, and further on we will keep running into it in places where you would not expect it: in memory, which rewrites a recollection every time it is accessed, in the environment, which leaves a quantum system a handful of stable states, in evolution, which spends billions of years trying things out and throws away almost everything it has made.
And in the machines we have been building for the last twenty years, too, while systematically denying them the right to a question: we feed them the whole picture, at uniform quality, down to the last pixel at the periphery, and then wonder why they do not see the obvious. We will take this up in the twelfth chapter, once enough material has accumulated; for now it is enough to note that a person who had his question taken away and was left with only the picture would fare no better.
Yarbus, incidentally, had nothing of the kind in mind. He worked on the physiology of vision and solved his own problems: how the jumps of the eye are organised, what happens to a motionless image, how the central zone of the retina works. Nobody at the Institute for Information Transmission Problems in the sixties was reasoning about computational budgets and the structure of observation.
This is, generally speaking, the best recommendation a result can have: data gathered for one question and continuing to work for another shows that you have stumbled on something beyond your own expectations. The experiments that fare worst are those set up from the very start to confirm a beautiful idea; they usually do confirm it.
Finally, a couple of words to keep from getting carried away. What follows from this is something more modest and more unpleasant than the hackneyed “people see only what they want to see”: of the canvas hanging in front of you, only a tiny fraction gets as far as detailed examination, selected by a rule you did not choose and are usually not aware of, and all the rest is filled in for you, and filled in superbly: the seam is invisible, and in a whole lifetime you have never once felt it.
The question this book is occupied with looks like this from Repin’s room. If observation is a request, then who formulates it and by what rule; what does it cost; and what happens to a system that has learned to put it to itself.
Let us begin with the cost, because it has been measured and is expressed in watts.