I've never had someone tell me that I'm scary, but I think I've just come close. A friend said that libertarians tend to scare her, and that oh yes, by the way, you (I, that is) seem like a libertarian. The scary part, it seems, is the fact that "libertarians seek to establish an order based on greed, want, and selfishness."
For the moment let that stand unsassailed, accepting that it is true. Does that admission inexorably lead to an unhappy conclusion? In the eyes of my interlocutor, it does, or if it isn't precisely poor, neither is it perfectly ideal. Yet is it not possible that the object of my desire is good?
In other words, can I choose to selfishly pursue selfless ends because I want to. Selfishness might lead to me helping myself at the expense of others, but it might also lead me to sacrifice for others. Isn't it true that thousands of people every year risk or lay down their lives in order to help others? Civilian contractors in Iraq face grave risks, just like human rights workers in the Sudan, firefighters marching into a burning building, or police and military officers fighting to protect others.
We can't ignore the other factors in play here - rarely is it that such behavior is totally without benefit to the actor, but it hardly seems to tarnish it. A just society very well may find itself grounded in selfishness, full of people tirelessly sacrificing for others, providing to them things that they wish to have at their own expense, assuming their own risks. They may do so because they profit, and so too may they do this because they like it.
As someone who calls himself a libertarian, I think things are just fine that way.