I’m going to play devil’s advocate here. Don’t take this the wrong way.
Not a problem when majority population actually understands Taiwanese.
As a blue collar worker, he may not have had a proper education and could only communicate in that dialect. He also could’ve been lazy and decided to skip mandarin because…buses are mostly used by locals.
I wouldn’t expect the older generation to speak mandarin. I’ve bumped into a lot of ah bei and ah ma in the stores and no matter how much mandarin I speak to them, they will respond and talk to me in Taiwanese. Don’t take it as them looking down upon you or something along those lines, they’re just used to speaking that language.
In lieu of that, they may keep speaking it to you in hopes that you may pick up something. Have you?
They have been speaking Taiwanese at family gatherings for as long as they could remember. Yes, it would be nice of them to consider your presence, but some things are hard to change.
I understand your annoyance, but maybe it would be better to adapt than to sit there and act clueless. If you take some time out of your day from taking care of the kid (kids?) or while the kids are at school to learn some phrases and use them in front of family, I am sure the family will be pretty excited that you’ve shown some interest and maybe want to learn.
You just have to be patient and have an open mind.