From an article about how some people really don't want to do breakfast/lunch/dinner with you to do meetings or because you want to "pick their brain." The fact that you are paying for the food is basically nothing to them.
Ask them out, and they won't just turn you down, they'll humiliate you for asking: “I don’t do lunch, except with my family or friends. We have very little time together, so I save down time for them." That's how a female career consultant — who'd just given a speech about women in business — reacted when a woman said "I'd love to have lunch with you and pick your brain."
Okay, so you need to preserve your time. Why can't you say that would be very nice but I can't fit it on my calendar? You don't have to announce that other people are more important to you. What is that? Is it preening over the fullness of your personal life? An attempt to educate everyone to think business-related socializing is now considered offensive? If you're so keen on withholding personal time, why don't you also care about withholding personal information?