The biggest differences you will see are in the speed you can travel and how nice an aluminum boat is to tow behind your rig.
The smaller differences you will see is that aluminum boats are easier to clean, easier to maintain, and tougher to begin with... I'd much rather hit a deadhead in my boat than in a Bayliner.
I've got an 19' Custom Weld, 115hp outboard prop, it's got an 11 degree vee at the stern. I normally take lots of gear with me, and loaded it will cruise at 35mph, maybe 40mph if the water is calm and I trim it out.
When I first bought the boat, I thought, "damn, this is a rough ride", but the more I ride in other (fiberglass) boats, I realize that it's not that bad in comparison, and in most conditions, (like that summer afternoon wind slop) those fiberglass boats would ride just as rough if they could go 35mph or 40mph. In my boat, if it's rough, I'll slow down to 15 or 20mph... it feels like you're crawling, because you're used to going 35mph... but then you see the guy next to you in the Glasply, and he's going 15 or 20mph, too.
I've had my boat in 10 to 12 foot swells, I've had it 7 miles out off of Vancouver Island, I feel very safe in it. I bought my boat used, and if I had to do it all over again, I'd buy it again...
You should figure out the other stuff, like will you be towing it a bunch? How much will your total boat/trailer combo weigh, and what mileage you will get towing it? Also, what sort of fuel economy will you get in either boat? The aluminum boats also cost a bunch more, but they don't depreciate near as quickly.
Now if we're talking new boats, well, I want to test drive an aluminum cat, just to see what those handle like...
-N.