Maybe it is my age, and my upbringing, but certain behaviors indicate to me a lack of class. And this has nothing to do with money, or wealth, or education. Rich or poor, educated or not, parents can instill or not instill certain cultural practices into their children as representative of what shows respect for others and for community values. Wearing a hat indoors is just one of those practices that I can't help but notice. Women with multiple tattoos, and for that matter men with the same, denote a certain degree of class, as does multiple face and other body piercings. Everyone is free to do as much of this as they desire, but it does send signals to the rest of society even if some fail to respond to those signals.
And just to show that I am not obsessed with men wearing hats indoors, many other ways of living denote the same degree of class (and self respect in my opinion) that show my upbringing. A front yard filled with junk, the use of excessive profanity in public regardless of the presence of women and children, are just some that come to mind right now.