conservative coworker keeps making "those swinger neighbors of mine" jokes. paranoia or is this an actual signal hes outing me

i go back and forth on this every wk so im posting. i work in a 22 person engineering firm. small. m41, married, lifestyle 3 yrs, very careful, no overlap btwn work life and lifestyle life. one coworker — call him brad — is mid 50s, openly conservative, "the country is going to hell" energy, makes a lot of comments abt "society today" that the rest of us mostly ignore politely. starting maybe 4 months ago brad started telling stories at lunch about "his swinger neighbors". the stories go like this — every 2-3 weeks at the wed team lunch, brad casually drops something like "oh you wont believe what those swinger ppl down the street did this weekend". sometimes its just a comment abt parking ("they had so many cars on a sat night, that swinger crowd"). sometimes more graphic ("you should see what they let people see thru their windows"). its always specific enough to feel real and vague enough to be plausibly his neighbors. most ppl at the table laugh uncomfortably or change the subject. i laugh along, badly, and feel like im being looked at while he tells the story. or am i imagining that. here is what makes me wonder if hes signaling vs venting. brad doesnt have neighbors. i know this bc i did the petty thing of checking 6 months ago — he posts on facebook from a house out in the country on a private road, his nearest neighbor is half a mile away. there are no "swinger neighbors". he is inventing them. that is the part that keeps me up. so im running this past myself constantly. maybe hes just a guy who thinks lifestyle ppl are funny and invents stories to entertain himself, probably the most likely honestly bc ppl invent neighbor stories all the time, its called being old. but then also maybe he overheard something abt me from somewhere (where?? im so careful) and the stories are designed to see if i flinch, i have no idea HOW he could have heard but i cant rule it out bc the lifestyle community in our city is small. or this is the weirdest possibility — maybe he is in the lifestyle himself and the swinger neighbors stories are coded venting about his own situation and i am the projected audience he doesnt realize he is performing for. would be wild but ive seen it before, closeted ppl create fictional 3rd parties to talk about themselves. m41, in a senior position at this firm but not over brad, our jobs dont intersect much but we eat at the same lunch room. i could ignore him completely if i wanted to. but the stories ARE about me adjacent topics, and the table dynamic is uncomfortable. what would you do. confront the stories. ignore them harder. change which day i go to lunch. read into the closeted-himself angle and see if hes secretly in this with me lol. i need an outside view bc im in my head abt it