I'm in Ouray now, and the hubbub in this coffee/chocolate shop is killing me. In terms of views from downtown, I'd have to say Ouray>Silverton>Lake City, but in terms of places I'd actually want to spend any time the order is exactly reversed. Lake City>Silverton>Ouray. In Lake City, hikers roll into town, and it's immediately Locals plus hikers versus Texans. The tourists do their ATV thing, but the kids also play volleyball and frisbee in the park. In Silverton, the park is on the far edge of town, and empty. The locals are cool, but the people most willing to hang out with hiker trash are themselves ski-bums instead of true locals. The Texans don't even spend the night, they just take the narrow gauge railroad up from Durango, crowd the sidewalks during the day, then leave on the last train. The jeep road out of town has more mines and less free car camping than that out of Lake City. Outside Ouray as far as I could tell it's totally private and you have to hit the fee-charging Forest Service site. The hostel in Lake City is owned by a hiker, and the Silverton one is definitely not.
That said, I ran the Bear Creek National Recreation Trail from Ouray up to Engineer Pass this morning and it was phenomenal. Up there with Eagle Creek on the PCT. And I'll be back in all these places whenever I get into Hard Rock.