When I was in first grade, my teacher once called the class together and said, “Glen is feeling bad because no one will play with him. Will anyone here play with Glen?” Glen was new (I think he moved to the area mid-year) and awkward-looking, and I remember how bad I felt for him when she said that. So I raised my hand. “I’ll play with Glen,” I said. To this day, I still remember the abashed smile he gave in response.

So we arranged a play date, and I went over to Glen’s house. And I discovered that Glen was nice — but boring. I don’t remember if he ever asked me to play again, but I do remember how uncomfortable I felt with the idea. And how even more uncomfortable I felt with the idea of telling him how uncomfortable I was with the idea. I didn’t want to hurt his feelings.

Nor did I want to hurt the feelings of the other peers I befriended in the years that followed who were like Glen: at the outer edge of most social groups, hungry for admission to any one of them. I not only felt badly for unpopular kids (not that I was so popular myself) but also outraged that some kids were popular while others were not. So I always lent everyone a sympathetic ear, and many took this as a signal that I was open to a friendship with them.

In some cases, however, I wasn’t. Though my goal as a Buddhist has always been to develop myself into someone who has compassion for everyone — that is, someone who cares about everyone’s happiness — my goal has never been to have a personal relationship with everyone for whom I feel compassion. For one thing, strange as it may sound, you don’t actually need to like someone to feel compassion for them — that is, to recognize their basic humanity or care if they’re suffering or not. What’s more, as hard as having compassion for everyone is, I think it’s actually easier than liking everyone. Compassion can be consciously cultivated. Preferences, in general, cannot. I don’t like every book I read, every song I hear, or every painting I view. So why would I expect myself to like every person I meet?

But flat-out rejecting someone’s friendship feels to most people too difficult despite the resentment we may feel toward others for thrusting themselves upon us as well as toward ourselves for our inability to express to them how we really feel. To reject someone romantically is hard enough. But to reject someone’s friendship seems to carry with it a uniquely harsh judgment, calling into question, as it may seem to, their value as a person.

But we’re no more in control of our attraction to friends than we are our attraction to lovers. And to reject someone as a friend isn’t to declare them unworthy of friendship any more than to reject them as a lover is to declare them unworthy of love. What’s more, some people value friendship more than others do — and further, the degree to which we value friendship changes as we age. (In adolescence, nothing seems more important. Then in middle-age our focus on friendship tends to decline as the importance of work and family increases. In old age, then, the importance of friendship may increase again as both the importance of work and availability of family diminishes.). We are who we are and shouldn’t criticize ourselves if we find we want to end a friendship. We’re not evil because we no longer like someone, or because we never did. Or never liked them as much as they like us.

Whether or not you should end a friendship lies beyond the scope of this post. But if you’ve decided you do want to end one, how should you do it? The way it’s been done to me (and how I’ve done it myself once or twice) is with what I call passive rejection — returning phone calls and emails sluggishly or not at all; claiming to be overwhelmingly busy or finding other excuses not to accept invitations — hoping all the while that in being prevented from engaging with us consistently that our friend will eventually lose interest in doing so. The advantage to passive rejection is that it avoids direct confrontation, thereby minimizing hurt feelings, as rare is the person who upon experiencing such passive rejection recognizes that his friendship is being rejected. (It just seems not to occur to most us.) On the other hand, passive rejection typically takes a while (sometimes a long while) and feels unpleasant. And it requires us to be dishonest.

The alternative, however, seems simply untenable for most of us. “Look, Glen, I just don’t want to be your friend,” I could have said all those years ago. But even then that seemed to me unforgivably cruel. Had Glen been in the habit of torturing small animals in his backyard, it would have been easy. But such obvious moral failings rarely represent the reason we want to end a friendship.

The truth is, though I’ve offered it here, and though it works, I’m not comfortable with passive rejection either. I hate lying in any form, including lying by omission. Further, because everyone knows this is how most of us do end friendships, when we turn down plans because we really are too busy our actions may easily be misinterpreted as attempts to end the friendship when they’re really not. This entire topic is uncomfortable, in fact, but I’ve observed enough people struggling with this issue to think it warranted discussion. When I began this post, I thought I’d come up with a satisfying answer. But having reached the end of it, I find I haven’t. So what do you think? What’s the best way to distance yourself from someone who wants to be closer to you than you want to be to them?

Alex Lickerman is an internal medicine physician who blogs at Happiness in this World.  He is the author of The Undefeated Mind: On the Science of Constructing an Indestructible Self.