Too bad we're not living in a martial art movie.
We're living in the real world.
Even if I had pummeled that guy, what's the consequence I would get?
A. His friends backed him up, and attacked me with some weapons.
B. None of his friends backed him up, but the police came and took both of us away, and my vacation was ruined.
C. No one backed him up, no police, I left the scene, but for fear of getting into troubles, I had no choice but to cancel my upcoming stays around that city, which also meant my vacation was ruined.
None of the consequences are tempting.