This is the classic situation of receiving the defining event after you need it in order to begin the action. Specifically, you can't know if the mouseDown is the beginning of a drag until after the drag starts. However, you want to act upon that mouseDown if a drag doesn't start.
In iOS (I realize that's not directly relevant to the code here, but it is instructional), there's an entire API built around letting iOS attempt to make these kinds of decisions for you. The entire Gesture system is based on the idea that the user starts to do something that might be one of many different actions, and thus needs to be resolved over time, possibly resulting in cancelled actions during the tracking period.
On OS X, we don't have many systems to help out with this, so if you have something that needs to handle a click and a drag differentially, you will need to defer your next action until a guard time has passed, and if that passes, you can perform the original action. In this case, you will likely want to do the following:
mouseDown, begin an
NSTimer set for an appropriate guard time (not so long that people will accidentally move the pointer, and not so short that you'll trigger before the user drags) in order to call you back later to trigger the popover.
mouseDragged, use a guard area to make sure that if the user just twitches a little, it doesn't count as a drag. This can be irritating, as it sometimes results in needing to drag something farther than it seems necessary in order to begin a drag, so you'll want to either find a magic constant somewhere, or do some experimentation. When the guard area is exceeded, then begin your legitimate drag operation by canceling the
[timer invalidate] and do your drag.
In the callback for the timer, display your popover. If the user dragged, the
NSTimer will have been invalidated, causing it not to fire, and so the popover won't be displayed.