I would reply. “Why the fuck would an Australian come here to work illegally and earn a crappy salary while risking deportation?” “Look at my fucking passport and you will see the visa dickhead.”
But I guess that’s why I never get invited to parties.