I swear this must be a Taipei thing. I live in Kaohsiung and I get spoken to in Chinese basicly everywhere I go. My Chinese sucks too (ordering food and taxi cab level).
The conversation will usually start in Chinese, then if they see me floundering they might switch to English if they can use it, or just look at me like I’m retarded and then start speaking more slowly.
Apart from the odd strange person wanting to practice their crappy English, it is mostly just young people with a strong command of English talking to me in bars or nightclubs or outside 7-11. Their English way surpasses my Chinese so it makes sense we would talk in English.
I have never understood why people get annoyed by having English spoken to you. You live in Taiwan, there are so many chances for you to speak Chinese, why get upset when a few people want or feel the need to speak English to you.
A Taiwanese girl was telling me how a foreigner got angry with her at a coffee shop when she tried to serve coffee to him in English. He said he only wanted her to speak to him in Chinese. Apparently his Chinese wasn’t very good either and she had difficulty understanding him.
I have had one old white guy who lives in my building let off a bunch of rapid fire Chinese at me just for saying good morning to him in English. I also say good morning to the Taiwanese in my building in Chinese and none of them act all weird about that.
Some foreigners in Taiwan are strange.