You shouldn’t…usually you should get worried if it is the other way…whereby they say that everything will be alright and let it at that…if they stamp it and sign it then it should be ok…but what was the stamp and what was signed on your passport?
You must remember that this was a mistake on the part of the officer at passport control and not yours. It is his/her job to follow the guidelines and laws for visas and for giving the correct number of days of entry that can be allowed with a certain type of visa, and not yours to have to correct them and tell them; and then suffer for their screw up.
If you wanted to, you could probably make a bigger deal of it with Immigration Control and perhaps get that passport control officer sent back to class to take Passport Control 101.
Or make it an issue with Immigration as another injustice against the foreigner in Taiwan, as their Passport officer seems either to misunderstand the laws and regulations to do with foreigners entering Taiwan or as in many cases with Taiwanese officials in many government departments, give their own interpretation of the laws and regulations