The Straight Dope chimed in on this topic recently, also comes down on the side of hand dryers.

Enter Una and her spreadsheet to take an objective look. With everything she could think of accounted for (admittedly different assumptions might produce different results), Una's best guess is that standard hot-air dryers use 5 percent less energy than paper towels in the first year, and about 20 percent less over five years. If high-efficiency dryers like the Airblade really provided acceptable drying in ten seconds, then they'd use 80 percent less energy... Electric dryers might be better for the environment in other ways, too. Assuming the same five-year life span, Environmental Resources Management estimates paper towel production generates 35 percent more acid rain and 286 percent more greenhouse gas emissions... What about those reusable cloth towels mounted on a big roll? An EPA study found that continuous cloth towels, as they're known, have a low environmental impact, requiring only about 8 percent as much energy as paper towels and about 13 percent as much water (including what's needed for manufacturing and laundering).

On the other hand...

(a) it typically took 43 seconds [to dry hands] under a dryer to dry as well as 12 seconds with a paper towel, but (b) most dryer users lose patience well before then, and (c) about 40 percent end up wiping their hands on their pants.

I thought about that 40% remark when I was drying my hands under the jet-blast dryer installed at the US Green Building Council's restroom.