American with Taiwan born Parent: TARC, NWOHR, NHI, Dual Citizenship, Double Taxation,

How does the Mainland Travel Permit for Taiwan Residents work when visiting the Mainland with dual citizenship?

Does the mainland only factor in the Travel Permit even if I hold a U.S. passport? I have a Chinese visa in my US passport too.