It’s all about control.
God doesn’t care about sexual purity but he does hate adultery.
Adultery isn’t necessarily sexual. If both you and your wife decided that having sex with others are fine, doesn’t affect your marriage in the very least (marriage is a contract), then it’s not adultery. How the two of you want to go about this is between you two, but it would be wise to include proper protections and STD testing, as well as steps to prevent unwanted children and all that.
Adultery can be spiritual as well, and it’s mostly betrayal and infidelity. God hates that.
But religious people don’t think in that terms, they see control of sex as a means of controlling people. If you can shame people into thinking that sex is dirty, you can basically control them, their relationship. It’s toxic as hell.
It’s very possible to commit adultery without having sexual relationship with anyone. Like if you decide to shut your wife out, and make decision behind her back, without including her, that is detrimental to her (and your relationship to her), that’s adultery. No sex needed.
Nothing in the old testament suggests that fornication is bad, considering that a bunch of kings of Israel seems to have mistresses and concubines.
Good example of adultery would be what King David did. He found a woman attractive, sent her husband on a suicide mission to get him out of the picture, then just took her as his own “wife”.
God was not pleased with that.
Adultery is a severe breach of trust. That’s what’s sinful. If David actually went and talked with the couple, and both the wife and the husband says “cool, we’re both fine with that”, God probably wouldn’t mind it at all, especially if David would care for any resultant children, and the couple would have presumably gotten a handsome payment for all this as well.