I’m not sure about your line of reasoning. . .
(1)
Throughout English (and European) history Jews were widely and rightly reviled for their custom of child blood sacrifice. It was seen as a unique and defining character of Judaism, and did the religion no favors in terms of PR. There were even occasional (and presumably hysterical and unfounded) accounts of Jews kidnapping children to mutilate. While there may have been an element of anti-Jewish prejudice to all this, it was not simply about prejudice. The public responded with similar revulsion when Islamic rulers forced genital mutilation on English prisoners.
Judaism and genital mutilation have been associated in the European public consciousness for centuries. That’s natural enough given that from the Reconquista until the Post-WWII period of more diverse immigration, Jews were virtually the only people in the Western World practicing genital mutilation.
My own experience has certainly been that when you criticize (male) genital mutilation, you promptly get labeled Anti-Semitic. You even get threatened with all kinds of things - e.g. I had a professor of legal history threaten to throw me out of his class for ‘anti-Semitism’ merely for suggesting a parallel between cultural practices involving the sexual mutilation of male and female children.
Actually I suppose the truth is that I am anti-Semitic in so far as I think Jewish practitioners of child mutilation should be jailed - obviously only after being given fair warning to stop. Plenty of existing laws could be used to stamp it out if western society would only get over its fear of offending minorities. Judaism should not be treated with kid gloves based on things done to it by a regime destroyed 70 years ago.
(2)
I’d guess that average person looks at a swastika and remembers the Jewish victims of the Holocaust, but forgets the Slavs, gypsies, homosexuals, mentally disabled, Jehovah’s Witnesses, Catholics, Freemasons, etc.
My point is that for most people the symbolism attached to the Swastika involves considerable simplification/distortion of reality.