published in Videre: Journal of Computer Vision Research, 1(1):2-19, 1997.
Synopsis: The article advocates and demonstrates a new paradigm for the computational study of active vision and sensorimotor control. Animat vision eschews physical robots in favor of artificial animals - realistic, zoomimetic virtual agents situated in simulated physical worlds. The work draws from the fields of computer vision, computer graphics, and artificial life.
Notes: Click on any image in this document to view a larger version. Try free-fusing the stereo images and stereo MPEG () movies by crossing your eyes.
The artificial fishes in their physics-based
virtual world as it appears to an underwater observer. The 3 reddish
fish (center) are engaged in mating behavior, the greenish fish (upper
right) is a predator hunting for small prey, the remaining 3 fishes
are feeding on plankton (white dots). Seaweeds grow from the ocean
bed and sway in the current.
More images courtesy of Xiaoyuan Tu:
Here are (low-resolution) movie clips of artificial fishes:
movie clip from the animation ``Go Fish!'' by X. Tu, D. Terzopoulos, E. Fiume, in ACM SIGGRAPH Video Review Issue 91: SIGGRAPH'93 Electronic Theater, Anaheim, CA, August, 1993;
QuickTime movie including clips from ``Go Fish!'' and from the animation ``A National Geo-Graphics Society Special: The Undersea World of Jack Cousto'' by X. Tu, D. Terzopoulos, R. Grzeszczuk, presented at the SIGGRAPH'95 Electronic Theater Evening Program, Los Angeles, CA, August, 1995.
fish animat is equipped with a pair of mobile eyes and is capable of
color, binocular vision. The stereoscopic view at the right is
acquired by the animat as it gazes at a red fish in the distance.
The animat has foveated vision. Its retinas
have a 64x64 high-acuity fovea surrounded by three annular peripheral
regions with progressively diminishing resolution (hence the
increasing image blockiness towards the periphery). The picture to
the left reveals the structure of a retinal image by demarcating the
peripheral regions with white borders.
The task of vision-guided navigation in a dynamic environment serves to illustrate the active vision abilities of the fish animat. Despite their limited quality, MPEG movies can better convey the dynamic scenarios. Here is an movie demonstrating the pursuit of the red target fish by the observer animat (centered in the video frame) and an movie of the moving target as seen by the eyes of the pursuer. The red fish tries to get away, but the observer can maintain a hot pursuit by keeping the target foveated and using controlled muscle actions to steer the body in accordance with the gaze direction. These MPEGs give an idea of the performance of early versions of the animat vision algorithms.
at the right shows the format of the ensuing MPEG movies which are
more recent and better demonstrate the robustness of the animat vision
system (note: these large 640x480 MPEGs will run at less than the
normal frame rate on most computers). The top half of the frame shows
the left and right retinal images of the observer animat. At the lower
left is a side view of the action, with the observer at the center of
the frame. At the lower right is a top view of the action. The line
emanating from each eye of the observer indicates the direction of the
gaze. The line is colored white out to the estimated range of the
foveated target and yellow thereafter.
The next demonstration shows an observer animat that first detects and foveates a reddish fish and then applies visually-guided navigation to swim towards this target fish and follow it. Here is an movie demonstrating active-vision-based sensorimotor control in the animat.
The retinal image stabilization process of the animat vision system implements an optokinetic reflex for the eye muscles that maintains the stability of the visual world. To demonstrate the difference between stabilized and unstabilized vision during locomotion, let's make the observer animat strabismic by locking the gaze angle of its left eye, while the right eye continually performs saccadic eye movements under control of the optical flow based retinal image stabilization process. Here is an movie demonstrating a fish animat with an inactive left eye and a right eye that actively compensates for the pitching and yawing of the swimming body.
We are extending the animat vision system to support other behaviors in addition to the pursuit of moving targets. We have implemented a module that integrates motion and color for selective attention. For example, the next demonstration shows an observer animat foveating and tracking a red target fish. The observer is also alert to interesting colorful objects moving in its visual periphery. Once it detects another target worth looking at, in this case a bluish target fish that enters its peripheral vision, it foveates the recognized object to obtain a high resolution view and recognize the target. This perceptual processing enables the observer to produce the appropriate behavioral response; for example, if the detected object is food, the animat pursues, or if the object is a predator, the animat attempts escape. Here is an movie demonstrating selective attention. The observer shifts attention from the reddish target to the bluish target at frame number 167 and pursues the new target.
vision system can now also support predation and feeding behavior.
The following movies show a large bluish predator fish employing
active vision to detect and foveate a small reddish prey fish and swim
towards the moving prey using visually-guided navigation. As the
picture on the right shows, when the predator manages to get
sufficiently close to the prey, it violently sucks the prey into a
gaping mouth and devours it. Here is an movie demonstrating the vision-guided predation
behavior, as well as an
movie showing what the predator sees during the predation behavior