Vanilla wife, lifestyle husband — five years in and were both miserable

Im Sinead, [F39], married twelve years. Before me my husband was in the lifestyle two years openly, clubs, parties, the lot. When we got serious he said he could put it down for me. Promised. Meant it I think. First three years grand. Then bit by bit he started bringing up fantasies again, mentioning a friend from his old club, mentioning a couple whod be up for it. I listen and my body shrinks. I have no desire to experience any of it. Never have. He says he loves me but this part of him doesnt go away. I say I signed up for all of you, not parts of you. The compromise seems impossible — either we go and I break, or we dont and I take something away from him that was his. Anyone solved this kind of couple? Is there a middle path that doesnt have one of us selling out? Or is it honest to admit different needs mean different lives?