Honesty as to "why" can go a long way.... Be honest with the person if you feel like you need to, and be polite (this is mostly because there enough rude people in the world) .... If they can't take no for an answer, do you really want them in your SL hounding you for whatever reason?
I use mine more as a contact list, but a few on my list are actual friends, the folks that make my SL fun.
I absolutely hate the "we talked for 30m, I'm a guy, you're a girl, let's be friends so I can try and stalk you /bootycall" invites though. Those people need to at least try to get some common sense and manners....