This is something he wrote in one of his letters (collected in the old Arkham House Selected Letters books):

“If I were ever to try to be a ladies’ man, I fear I would upset all my charmers at the outset by weaving them into weird and horrible tales! But fortunately, horror writers are not often ladies’ men.”