Okay... I figured it out, and I'll post the solution here in case someone else needs the reference later.
I transform each point as follows:
Public Function TransformPaperspacePointToWCS(ByVal oViewport As Viewport, ByVal oPointToTransform As Point3d) As Point3d
Dim oRotationMatrix As Matrix3d
oRotationMatrix = Matrix3d.WorldToPlane(New Plane(oViewport.ViewTarget, oViewport.ViewDirection)) * Matrix3d.Rotation(oViewport.TwistAngle, Vector3d.ZAxis, Point3d.Origin)
Return oPointToTransform.TransformBy(oRotationMatrix.Inverse())
End Function
This only works with viewports that have been rotated around the Z access which is typical on our plan sheets.
Next, I need to figure out how to collect the paperspace points from irregular shaped viewports.