There should be two counters one for karma and other for anti-karma. People with positive net karma couldn't downvote people with negative net positive karma and vice versa. This should reflect two kind of peoples. I should prefer to be in top of anti-karma.