I just want to reiterate that the word Huan doesn’t necessary mean barbarian in the Latin sense.
I mean consider these usages:
一番
一番話
三番兩次
番號
The word and character are ancient, and it originally meant to do things in turns, to flip over things (in turns), or to dig up soil (in turns as in one after another). Original character means imprints in the farm. Its derived meanings gave us characters like 翻, and 播. The meaning of in turns and one after another gave it a derived meaning of numbers, or groups, and divisions.
The first usages of the character to refer to non-Han people seems to be referring to Muslims. 入番 meant converting to Islam, and 番僧 meant Imams. This usage is no earlier than the Yuan dynasty, in fact the earliest usage I can found are all from Mid to Late Ming. After the term was popularized it was also applied to Europeans and retroactively to non-Han people in the past.
Since the character never had the meaning of barbarians prior to that point, my guess is the word is a phonetic translation for a foreign word used to describe foreign people, and it was adopted as a way to be polite since it would have avoided using words like 蠻 or 夷, which meant barbarians at the time. Since neither the Muslims nor the Europeans were really barbarians, after all they had ships that could travel to China to conduct business, the first usage of 番 in the non-Han context was just “foreign”. Words like 番茄 (tomato), 番石榴 (guava) all reflect that. I mean foreign eggplants and foreign pomegranates make a lot more sense than barbaric eggplants or barbaric pomegranates.
My personal theory is that 番 would be the phonetic translation of the Persian/Arab word Farang, which was their phonetic translation of the word Franks, and used to refer to Europeans in general.
I think the context was still foreign when it was first used to refer to Indigenous Taiwanese people, like 東番記 (Foreigners to the East) written in the 1603 about a 20-day military expedition to drive out pirates based in Taiwan.
If 番 indeed was a phonetic translation of the word Farang, then it has come full circle when it is again used to refer to white Europeans.