People
Well, no, not married as such, but yes, there is a specific girl that I'm not married to.
— Douglas Adams ‧ Dirk Gently's Holistic Detective Agency
Behind the line May contain spoilers.
Our reading
An editorial interpretation.
A funny bit of hair-splitting: the speaker rejects 'married' but confirms there's one particular girl they aren't married to. It's an answer that technically says yes while sounding evasive, which is where the joke comes from.
Context reviewed