The view from the TV tower (or whatever it is) on the central “mountain” (100-200m high :p) is really good:
The last hundred or so meters of road that leads there are off-limits. But honestly, if you can’t read the smallish Chinese sign saying so (I couldn’t), there is nothing else that would indicate so. So I guess just check if there are any workers etc. around, it looks pretty safe to go there. It is not military, after all.
Anyway, the walk over the central “mountain” is nice even if you don’t go to that tower.
The one thing I would suggest doing, even if you are not so much interested in the military history of that place, is going to the knife factory of “Maestro Wu”. It’s just impressive to see those hundreds of artillery shells (don’t worry, all inert - propaganda shells, not explosive), and also fun to watch how they are cut and made into knifes by hand. The knifes are very affordable, look good IMHO, and are very easy to keep really sharp: in my kitchen even my Japanese Kyocera ceramic knife had to make way for the few-hundred-NT$-artillery-shell-knife from Kinmen.
For the military stuff (even I am quite interested in that) I would give almost everything a miss… There is a nice, big museum with lots of heavy equipment somewhere is the western/central area (edit: That should be the “Rushan Old Barrack”).
One more thing I liked is the smallish observation bunker museum here:
When you go though the tunnel to the observation post, you can use the binoculars there to see the Chinese fishermen just ~2km away pick their noses Also if I remember correctly, right in front of that observation post there is a little rock island which has been fortified to the extreme - a nice look into the past.
In the village Jincheng there is the probably only bar worth mentioning on this islandn named Bar Sa. Cute communist propaganda decoration, and they offer the only way in which I think Gaoliang liquor is enjoyable: Their “Kinmen Battlefield” series of cocktails, you can read more about it in this thesis paper on pages 104 ff.
Oh, and skip Little Kinmen Taking a day trip by ferry to Xiamen /Amoy (China) can be interesting, if you want to see a bit of the mainland. It’s not very different than Taiwan anyway, except for bored policemen hanging out at every street corner.
Maybe I will add some more info later when I have time…