Nonphotorealistic Rendering

Short Summary
Project Goals
Discussion
Open Questions
Papers

Summary

Our nonphotorealistic rendering work has ranged from the imitation of classic media like pen-and-ink or crayon, to algorithms for rapidly extracting information (like contours) from models, to the study of higher-level perceptual issues like composition.

Project goals

The goal of nonphotorealistic rendering (or, as some of us prefer to call it, expressive rendering) is to empower the user to depict what matters, and to do it in whatever way s/he may please, rather than in the way that the motions of photons might predetermine.

The "controllability" aspect is reasonably well under control, I believe, but the problem of knowing what to render is still quite open. In particular, to know what to render, it's important to understand something about human perception. We don't know, for instance, exactly which lines to draw to best communicate a shape. The silhouette -- dividing the shape from the background -- is important, as are the contours -- points where the view direction is tangent to the surface. But there's evidence that even within these curves, there are things that a more and less important: it's easy to recognize many shapes from a contour drawing in which only the 20 percent of the contour with greatest planar curvature (in the image plane) is shown, for instance.

Recently, researchers at Rutgers and Princeton described "suggestive contours" as a possible important topic of study; I believe that these have the potential to be quite important -- far more important, for example, than ridges and valley curves, which have been extensively studied.

My current goals are to better understand suggestive contours, and their relation to the human visual system and perception.

Discussion

In the early days, NPR was all about imitating media effectively; we were learning, through imitation (a classic technique in art class!), how various things were done by artists, and how to simulate them, at some level. Unfortunately, the simulations were often not very good: we made pen-and-ink drawings that looked like pen-and-ink drawings, unless you knew something about those, in which case they looked like amateurish pen-and-ink drawings, and so on.

Things have moved on. There are still more media to simulate, and papers about that will continue to appear. But now we've begun to consider the WHY of certain artistic techniques: why do artists draw outlines? Why do they always draw shadows, but so often draw them approximately rather than carefully? What choices do they make in deciding how to expend effort in making an image?

One aspect of NPR in which I firmly believe (although I do not know how to prove it) is that informal output is better suited to the design stage of a modeling task (be it architecture or video-game-character-creation) than is precise a photorealistic rendering. Indeed, much of my interest in NPR comes from my interest in sketch-based modeling, to which I believe it's intimately related (although reviewers of my papers have sometimes expressed grave doubts, which I must take seriously).

I'm currently intrigued by a particular effect that artists use sometimes: the omission of regions. I can, for example, draw a torso, and simply draw the sides of the neck, with those lines fading out; you, on seeing the picture, don't think "What happened to the head?" You understand that the head is there, but not drawn, or not important. If I had terminated the drawing by putting an oval between the two lines, indicating the neck had been severed, that would be quite disturbing by contrast. What are the techniques that work and don't work for indicating intentional omission?

Open Questions

There are lots of open questions in NPR. A few of my favorites are:

Why are most points of the suggestive contour generator on the back of an object?
"The main reason why line drawings are so efficient [at depicting shape] is because they represent loci ob objects where view-dependent surface features exhibit a discontinuity in image space," says Pascal Barla. Is this true? And if so, how do we reconcile it with the power of suggestive contours?
Is the "temporal coherence" problem in NPR simply a mistake? That is to say, are there perceptual reasons why animated 2D "drawings" will always create noticeable artifacts that distract us from the intended perception of the motion? Shakespeare may have been on to something in Troilus and Cressida when he wrote that "... things in motion sooner catch the eye than what stirs not."

Papers