Assuming your polyline is 2D, you can use (vla-getboundingbox ....) method to find out the bounding box of the polyline.
Then you can do (ssget "_CP" (list Pt1 Pt2 Pt3 Pt4) (list (cons 0 "INSERT"))) .... of all block entities within this bounding box.
where Pt1, Pt2, Pt3 and Pt4 are the vertices of this bounding box. Note to do (vla-zoomwindow (vlax-get-acad-object) (vlax-3d-point pt1) (vlax-3d-point pt3)) before you do above ssget because it is important that the vertices of the polygon within which you want to get entities must be zoomed in on screen. Here Pt1 and Pt3 are (minx, miny) and (maxx, maxy) coordinates of the bounding box.
Then you can iterate through all the objects to find their insertion point and check whether the point lies on the polyline or not using (vlax-curve-getDistAtPoint..... method.