Why are people judged by the way they look?

Sometimes you may wear a funky looking shirt or someone may say something about what you are wearing. But when I say the way that they look I mean there color. There are many times in History where people were kicked or put of their town or discriminated against. When I say discriminated i think of the times when there was racism. There is many times in History where people were discriminated against.