Seems too much like work, to me. I dont see the need to convert anything; and Im not sure why your routine isnt working - are you selecting a single line as an object? Because a single line, or a polyline with only one subentity, will only have 2 points (coordinates), which isnt enough points to describe the polygon you wish to select by.
Also, you realize that in a lot of cases, the polygon you wish to use to select by WONT also select, say, the polyline with the coordinates you used to describe the polygon? because the polygon would be ON, not IN, the selectionset polygonal window.
Try the code below. ts beasically whipped up from copy/paste from the help files; it creates a triangular polyline, and a circle inside it, then creates the selectionset boundary from the polyline coordinates. Note that the circle is selected, and not the polyline....
' -------- snip -------- snip -------- snip -------- snip --------
Sub Main()
' Create a polyline. Then use the Coordinates
' property to return all the coordinates in the polyline.
Dim plineObj As AcadPolyline
' Create Polyline resembling a triangle
Dim points(

As Double
points(0) = -10: points(1) = 0: points(2) = 0
points(3) = 10: points(4) = 0: points(5) = 0
points(6) = 0: points(7) = 7: points(

= 0
Set plineObj = ThisDrawing.ModelSpace.AddPolyline(points)
' Return all the coordinates of the polyline
Dim retCoord As Variant
retCoord = plineObj.Coordinates
' Draw a circle in the middle of the triangle
Dim MyCircle As AcadCircle
Dim CenterPoint(2) As Double
CenterPoint(0) = 0: CenterPoint(1) = 3: CenterPoint(2) = 0
Set MyCircle = ThisDrawing.ModelSpace.AddCircle(CenterPoint, 1)
ThisDrawing.Regen acActiveViewport
' create a selection set
Dim mode As Integer
mode = acSelectionSetWindowPolygon
Dim mySelectionSet As AcadSelectionSet
Set mySelectionSet = ThisDrawing.SelectionSets.Add("TEST_SSET")
' use our points from the polyline to select entities
mySelectionSet.SelectByPolygon mode, retCoord
If mySelectionSet.Count > -1 Then
Dim acEnt As AcadObject
For i = 0 To mySelectionSet.Count - 1
Set acEnt = mySelectionSet.Item(i)
MsgBox "Item " & i & " " & acEnt.ObjectName
Next i
Else
MsgBox "Nothing in selection set"
End If
mySelectionSet.Delete
Set mySelectionSet = Nothing
End Sub