telling another play partner "my spouse is primary, you're not" — fair labeling or actually kinda gross

wanna think out loud here bc this came up between dh and me last weekend and we couldn't land on the same answer. context, were F37 M40, LS for 5 yrs, full swap currently, mostly couples but i've started seeing a single male friend (vetted thru a friend, very safe, dh has met him, all that). nothing romantic just a fun fwb situation roughly once a month. the issue. last meet, he asked something about a holiday we're taking and dh said in passing "well shes my wife so obvs her trip with me comes first" and the guy was a bit weird about it for the rest of the night. afterwards dh and i talked and dh said "i wasn't being possessive i was just stating reality, hes not my equivalent". and i agreed in the moment but ive been turning it over for a week. bc on one hand, yeah, dh IS primary. were married. we share a mortgage and a kid. its just facts. labeling those facts isn't a power move, its a description. but on the other hand. solopoly people online write about hierarchy as if any hierarchical labeling is inherently disrespectful to non-primary partners. like, the act of saying "you are lower in my structure" is what does the damage. and our guy clearly heard it that way. so was dh "just stating reality" or was he putting a smaller person in a box they didn't agree to. is hierarchy in ENM honest organising, or is it primary couples using language to keep secondaries small. how do you guys handle this when its a recurring play partner not just a stranger.