I hadn’t been to Tamshui in a year or so and was SHOCKED to see how nice it is–beautiful new boardwalk, outdoor cafes. You can walk from the MRT all the way to the end (keep walking until it forces you back inland. Check out Hongmaocheng (Fort San Domingo) which is open until 5pm (and cheap too). Then head back to the boardwalk where you can stay for the sunset or:
Take the ferry across the Tanshui River to Bali–there’s a great boardwalk there too!! Also brand new. Walk toward the visitors center, from there you can take a bus to the Shih-san-hang Museum which is also (yes!) brand new, and beautifully constructed. It’s an interesting explanation of one of Taiwan’s largest and most fascinating archaelogical digs. It’s open until 6pm and has a coffee shop on it’s back which is open 'til 7 (part of the museum is shaped like a sand dune… the coffee shop perches right on top). You’ll want to get there by 4:30 at the latest, though, because the museum stops selling tix at 5:30 and you want enough time to walk around.
You can also take a bus to the (yes, brand new) fisherman’s wharf (yu ren ma tou) for a nice walk. It’s lit up at night.
If you want to make a whole day of it (which might be tough with a four-month-old), take the Tanshui MRT to Kuandu first, and a taxi to the Kuandu Temple (has a neat tunnel filled with statues that leads through the mountain to a beautiful lookout on the Tanshui River). You can walk from the MRT station, but it’s about 20 minutes. Someone mentioned the crystal museum–nice place for a tour and a coffee. It’s also a 10 minute walk from the Kuantu MRT station (you’ll need to cross over the tracks first).
One final note–Tamshui now has mounted police (I believe they are on the Bali side). They make appearances on Saturdays at 10 or 11am, and then again at 2-4 (I think??). I believe that they also interact with the crowds.
Most of the things I’ve mentioned are free or cheap, and the crowds are sparse during the week… I did the above in an easy half-day 1pm-6:30pm on a weekday (minus the crystal museum and fisherman’s wharf).
hope that helps!