Yes, we don’t know the details. You and most of the others here want to extrapolate and fill in your version that points to the Taiwanese people willingly speaking English until Peter forces everyone to change. I suggest there is another way of looking at the situation that challenges the likely worldview of the OP. I accept the reality; everyone there was free to speak whatever language they wanted - if you believe that the Taiwanese had the choice as to what language they wanted to speak notwithstanding OP’s presence, I don’t see why you think they suddenly didn’t have such freedom after Peter’s comment.
You mention Peter’s supposed comment on how “america is so racist towards asian people” and “Eddie Huang.” Per the OP, Peter said that no one in America would ever switch from speaking English to Chinese for so long to accommodate a foreigner, and that it was unlikely he (an Asian guy) would get invited to hang out at all - you view this as “racism” but I imagine that most ABCs and foreigners view it as “normal” in the sense that we wouldn’t expect the dominant culture in America to accommodate or privilege us. (if ABC or other minorities are going to talk about American racism in the era of living people, the Japanese American internment, Jim Crow, the death of Vincent Chin, the persecution of Wen Ho Lee, Trump’s Mexican wall, and Black Lives Matter are going to come up first), Eddie Huang’s fame isn’t about Taiwan or “master of Asia” as you put above, it is about his experiences as an Asian American and growing up in America including racist experiences - not sure what the Taiwan connection is either.
I can see how such perspectives might challenge the worldview of those who would prefer to believe in an America free of racial privilege and discrimination, or a Taiwan free of privilege for white English speakers. But you reference the fact that Taiwanese sometimes pay Blacks and Asians less for teaching English - so you know that English and white privilege exist among Taiwanese. I invite you to consider that it exists in social as well as workplace contexts.