Another consideration is :
Do you want to do all this without user interaction
or would the user select the polyline and ( possibly ) select a window area for the candidate points ??
If candidates can be selected the process would be much simpler and faster.
>>> If there are 20000 points,Is the efficiency of this method too low?
For instance, this number could be reduced considerably.
If the process is a black box ( sight unseen, no interaction ) any arbitrary choice ( like an expanding selection area based on the polyline bounding box ) would probably be slower than a programatic selection and test of all points.
Though, I s'pose the complexity of buildibg a selection area around the polyline would depend on the shape of the polyline.
Knowing the usecase would assist in designing a solution [ the first consideration would be how often the routine is used ]
my gut says get it working first , then test if the speed seems acceptable for you before looking for optimisations.
Regards,