Thursday, June 21, 2018

Why Do Movies Move?

If you spend time watching movies or TV, you have probably know that you see a moving image on the screen, but that the sense of motion is created by your brain from a series of static images.  Typical movies, for example, flash 24 frames per second.  Somehow, the brain takes the changes from one frame to the next and gives you the illusion of fluid movement. 
How does that happen?  Take a moment to try to explain it to yourself.
This question is just one of many that is explored in a great book that came out in 2015 called Flicker by cognitive neuroscientist Jeff Zacks.  The book itself explores a variety of topics ranging from low-level aspects of the way the visual system understands the images on the screen all the way to high-level topics like the reason why movies are so good at creating emotion.
So, why do movies move?
You might think that what is happening is that each image persists a little on the retina (the cells at the back of your eye that respond to light) and that changes in the image are detected there.  Or perhaps, when the image is first processed in the brain, it recognizes small discrepancies from one frame to the next.
Neither of those possibilities is quite right.
The images from the screen enter the eye and hit the retina.  From there, they are passed into the brain and ultimately make their way to the occipital lobe in the back of the brain where most visual processing is done.  Initially, the brain looks for simple visual features in the image like the presence of edges, because edges usually signal the boundaries of objects.
The interesting thing is, the brain divides up the task of understanding the image in multiple ways, with different brain areas searching for different features in the image.  The unified sense of vision we have arises because the brain ultimately puts all of those independent properties back together. 
Early in the processing of images a particular area of the brain called area MT (shown in the figure) looks for blobs that have changed position.  When MT sees a blob in a location that has changed it position a bit, it gives a signal suggesting that there was motion.  Sustained activity of MT indicating motion in a particular direction gives people the experience that an object moved.  This brain area doesn’t really care much about the blobs themselves.  The blobs could actually change shape or color from one frame to the next, it is just looking for motion.
So, the motion in movies comes from activity in the brain area MT.  As Zacks points out, though, this can sometimes cause problems.  In particular, when a movie is put together, it is usually constructed from a set of scenes that are spliced (or cut) together.  If the editor is not careful, when one scene is cut to the next, some of the objects may appear to jump from one location on the screen to another.  This jarring sense of movement is called a jump cut.  Filmmakers try not to create these jump cuts, and texts on film making give suggestions for how to avoid them. 
These jump cuts are caused by the same process that causes the sense of motion in scenes.  When one scene is cut to another, if area MT detects the motion of a blob, it will send a signal that an object in one location actually moved to another.  That can feel weird, because brain areas that calculate the size, color, and shape of the object may not see strong similarities in the objects from one scene to the next, so you can get a feeling of motion without have a clear sense of what moved.
One of the reasons why Flicker is an interesting read is that Zacks explores the ways that movies exploit the structure of the brain to give us an immersive experience.  If you can pull yourself away from the screen for a few hours, it is well-worth the time to check it out.