IN THE COMMENTS: XWL writes:
For me, it boils down to, women read their porn, while men watch theirs.
If men got their 'jollies' (I mean erotic escapist fantasies) from books the way some women do, I'd bet those fiction/non-fiction gender differences numbers would level out.
According to this 2004 AP story, 40% of fiction book sales in the USA are romance novels.
Same article claims a remarkable leap of male readership from 7% in 2002 to 22% in 2004, citing Navy Seals as being big fans of a particular series of romance novels featuring Seals as one example of the changing type of romance fiction and the audiences they are attracting.
That jump in sales to men happens to correspond to a lot of young men being deployed overseas to places where the military prohibits possession of pornography (and alcohol, too, our troops are ordered to behave like better muslims than the people in the countries they're rebuilding). Real porn is prohibited and punishment meted out if found on base in Iraq, Afghanistan and Kuwait (though keeping young men from their porn seems ill advised and impossible in the internet and digital media age), those troops are forced to get their porn (at least their non-illicit porn) the way many women do, through novels.