The point returned by entsel is the location of the crosshairs (i.e. center of the pickbox) when the user clicked for the selection and hence may not lie on the selected entity. If the entity is a curve object, the simple solution is to use the vlax-curve-getclosestpointto function to find the closest point on the curve object from the point returned by entsel.
However, in your case, since the selected object is not a curve object and you require an accurate point selection, I would recommend instead using the getpoint function to allow the user to specify a point, and then select the entity located at the selected point using the nentselp function.