When I’m in Taiwan (I’m a white-faced westerner) I speak Chinese with almost everyone. For the most part it works just fine, and if I miss part of the conversation the people I’m with just expect me to keep up. It’s not often someone wants to use English with me.
Another factor might be one’s proficiency with Chinese. Even though I feel insecure in my Chinese speaking ability, I can say objectively that my listening comprehension is good, my pronunciation and tones are great (which tricks people into thinking I’m fluent), and I can read and write OK. In contrast, I know westerners who are married to Taiwanese and live in Taiwan for years, and their ability to speak and use the language is crap.
It’s painful to listen to a foreigner who butchers the language because they can’t distinguish tones or they speak in a monotone. In that case Chinese speakers will switch to English because you’re torturing their ears!