I’m from UK and came here with my Taiwanese wife 4-years ago. I found it difficult to settle in (still do I suppose). It does get better though, and you get more used to it too. I live in the north and I don’t know that it would be different up here. I think that when it gets into winter you’ll be glad to be based in the south.
In my experience some of that 85% group are a bit shy/nervous/afraid of you. They’ve not necessarily got any prejudice against you they might just be worried that their English isnt very good (although it might be better than your mandarin - don’t know you but my mandarin is ropey) and they don’t want to feel embarrassed if you talk to them and they don’t understand.
Advice - be outgoing when your out and about, try your best even if it’s not your natural personality type. You’ve got to make some first moves. And don’t feel held back if you’re Mandarin isnt up to scratch, people won’t expect you to speak Mandarin. Taiwanese people don’t generally go around smiling at people so the 5% that are might be people who are hoping you’ll engage/interact with them in some way. Take the time if you can to say hi, it’s a long shot that any will become lasting friends but maybe a few will and it might help your frame of mind.
Also, try and find some expat friends near where you live. I know that’s probably not what you came here for but it’s good to share some experiences over a beer or three.