I’ve never really had any issues with my merino base layer and that’s all I wear in cold weather. It could be your jersey or jacket isn’t breathable?
Typically, base layer is there to help regulate your body temp or whick sweat, however, if your outer layer isn’t really windproof, you are in for quite a cold descent.
For point of reference, on Xmas day, I had a merino base layer, long sleeve fleece layer and a thick wind breaker jacket and I was still freezing going down the hill.
One item I always bring is a helmet cover. While going up I store it somewhere.
But if it is raining, if it is getting dark, or when going downhill i put it always on, sometimes even in the summer.
Like this:
https://www.amazon.de/VAUDE-Helmet-Raincover-Wasserdichter-Helmüberzug/dp/B00ONW13ZC/ref=sr_1_4?keywords=Helmet%2BCover&qid=1703626284&s=sports&sr=1-4&th=1
Pro Tip for European winters:
Some winters ago I always wore my skiing helmet when riding to work during sub zero temps 
Just wanted to jump in here and say I have no problem myself with my own Merino layers. Are yours 100% or some kind of blend? I buy mine from a UK company called Vulpine…although they don’t apparently, currently, ship here (I asked). But if you can get them, they’re great. They wick like crazy. I’m ridiculously comfortable in them, whatever the weather.
For very cold temperature can try these under long finger gloves. Maybe there is also something similar for feet. Some plastic bag or something. Can take off easily if get too warm.