I’m sorry to break it to you, but I believe your boss was correct.
4/3 was meant to be a work day, but because it was adjacent to a holiday, it was swapped with 3/25 so that people can enjoy four days off in a row.
However, because 3/25 was already a regular work day for you (you are contracted to work Saturdays, no?), you would have had to work on Sunday 3/26 (or whatever) if you wanted to take 4/3 off.
You can’t just take 4/3 off without working an extra day.