Even Domestic, I carry mine. More commonly recognizable than a Drivers Licence.
I don't carry my paper passport around, but I do carry a US passport card in my wallet. It's come in very handy on a number of times and was only $15 when I renewed my passport. As an added benefit, it counts as a "Group 1" identification for employment verification and SIDA badge renewals.