If I object to being called a Brit does that make the caller a racist?

A lot of nationalities seem to object to a shortened version of their nationalities being used to describe them, Yet they call me a Brit