Yep, I have a few that are pretty hot and I've never jerked off fantasizing about them.
Sure, I've wondered how they look naked, and how they like their sex, but during usual moments, like watching them walk away and seeing their ass sway. But I've never specifically jerked it to thoughts of them

I never masturbate to women only men

No.

No, every female friend and most acquaintances have been fucked by me, in my mind anyway.

Yes and No!

I've known some very attractive women but they oddly didn't appeal to me in "that way".

Others - I have, some plenty of times. Really fancied some but had no chance with any of them.

Yes

You mean objectively attractive, or attractive to you?

If your friend is "I can see she is smart and quite pretty and etc. but I don't fancy her" then she is not attracting you.. quite likely you can not jerk to her.

I don't think you can have a female friend you fancy and not jerk to her, without special circumstances.

Its natural wanting to cum all over her or bust a nut in her pussy.