CPS prosecutor + LS, the conflict of interest scenarios keep me up

crown prosecutor, been with CPS about a decade, midlands. dh and i started LS about 2yrs ago. the conflict of interest thing isnt theoretical for me, like i prosecute sexual offences and other serious cases and the idea that i could one day open a file and recognise the defendant from a club is genuinely something i think about. obvs in that scenario i'd recuse instantly but the act of recusing creates its own questions, your line manager wants to know why etc. and beyond that specific scenario theres the broader question of whether being LS sits comfortably with prosecuting cases that involve sexual conduct context. has anyone in prosecution (or defence honestly) thought through this? not asking for legal advice clearly just want to know if other ppl in similar roles have wrestled with this and how u landed...