When testing it out myself, I didn’t see much of an uptick for a rarely visited site (like 20mb).

You can try for yourself, and see what you come up with. Here’s a tutorial: http://scottmotte.com/archives/145

I would recommends it for lightly visited sites for clients and things. I believe Ben Hughes and/or a commentor agrees on this at: http://www.railsgarden.com/2008/04/12/configurating-passenger-mod_rails-on-slicehost-with-ubuntu-710/