Being polite or wanting not to be embarrassed has kept people in situations many times that were not good for them at all. You have the right to not be in places you don’t want to be or with people you don’t want to be with. Walk away, don’t worry about it because you won’t be in that place or with those people again anyway.