In my experience…it depends. Just stick your card in when you get there and see what happens. I usually treat the numbers as a guideline and try to arrive earlier if the number is low and later if it’s high (although the meaning of “low” and “high” varies here - some clinics only seem to handle a couple of dozen patients per session, whereas for others it’s over a hundred and the sessions routinely overrun).
But I don’t worry too much about missing my number being called as every hospital I’ve been to (NTUH, Tri-Service, Mackay, Cathay, and TMUH) just adds you to the end of the current list anyway, and I’ve also seen high-number patients being called in early if nobody else is there. I actually prefer to arrive later, often after my number has already been called, to avoid the risk of having to wait around for hours. You can usually check the current number on the booking page and try to time your arrival if you like, at least for major hospitals.
It’s better to try to arrive before the clinic cutoff time for that session though (usually 30 or 60 min before the nominal end of the session and stated on the appointment booking page). Some hospitals/doctors are strict about that and other’s aren’t, but if you arrive after that you risk not being able to see the doctor (especially if they weren’t busy and have already rushed off).