Some people will treat you as an equal and with respect, others will look down on you and treat you like your opinion is worth less than theirs or others. Who you spend time with, make the effort to see and care about their opinion has an impact on how much you value yourself.
By choosing to spend time with those that treat you badly you are telling them and yourself that their behaviour is acceptable to you, this lowers your self-worth.