There has been a ton of discussion on this topic (maybe here but I know on the old VBDesign)
For simple cases where the polyline doesn't intersect itself and everything thing is in a plane with no fancy USC, one can count the number of times ray intersects the pline. An odd number of intersections means the point is inside, even number means the point is outside. The point falling on the polyline can impact the results too.
Here is some code that handles the simplest of the simple cases (assumes only 1 or 2 intersections possible)
(defun css_isPtInside(pPl pPt / oPl oPt1 oPt2 r1 saInters retVal )
;;returns T if pt is inside or nil if it isn't
(if (not (equal (type pPl) 'VLA-OBJECT))
(setq oPl (vlax-ename->vla-object pPl))
(setq oPl pPl)
)
(setq oPt1 (vlax-3d-point pPt)
oPt2 (vlax-3d-point (polar pPt 1.5 1.5))
r1 (vla-addray (vla-get-modelspace (vla-get-activedocument (vlax-get-acad-object))) opt1 opt2)
saInters (vlax-variant-value (vla-intersectwith oPl r1 0))
)
(if (equal (vlax-safearray-get-u-bound saInters 1) 2)
(setq retVal t)
(setq retVal nil)
)
(vla-delete r1)
(vlax-release-object r1)
retval
);defun css_isPtInside