No no no, Carter wasn't restricting his comments just to a few people holding signs he took as racist. That doesn't even make sense...what's the point in saying or implying that "the overwhelming portion of the intentional animosity displayed by the people holding racist signs is motivated by racism"?!
No, the press and the pundits on both sides and the administration too took that quote and a few other things he said at the time to refer to the whole demonstration and the movement associated with it. Carter made a gross generalization about a million people, and, by implication, many millions more like myself who were there in sympathy, based on his own preconceptions...that's called bigotry. And so plenty of educated, well-meaning people, some of them posting on this very thread, followed his lead and made accusations referring to the whole mass of demonstrators, not to a few particular ones.
Most people don't like to be called racists. People who love and respect their fellow man, regardless of their color, don't like being called racists. People who are afraid or unsure about those who look and act different don't like being called racist. Out and out racists don't like being called racists. And for decades it has been a very effective rhetorical tool for diverting arguments from issues, and for intimidating one's opponents. It's an ad hominem attack. As Sam points out, it is losing its potency, and becoming just a name thrown around indiscriminately. The best way to parry it is the way you'd parry any childish taunt. If someone calls you a racist, just call them a racist back.
Jimmy Carter, you're a racist.