Yes it's a bit more complex than that, because he's referring to...

Yes it's a bit more complex than that, because he's referring to a bad reputation as if it's a fact, and then he's also saying that this person is incapable or incompetent, by implication in the next sentence.
Taken together, these have the potential to negatively affect the subject's reputation.
He's also blaming the reputation as the sole cause of the SP.

What a lucky guy that most people are so tolerant so far. But apparently, I'm the baddie! I should stop right now!