The algorithm is flawed. See the breakout:
In this case it is the result of bounding dtheta, the removal of the two while statements would fix it. This is because by bounding that difference we are getting the smaller angle between the two vectors (which in this case are confusingly labeled pt1 and pt2) and because you are dealing with a simple convex polygon the angle will always be in the range [0,pi] or [-pi,0] depending on rotation direction of points, and the sum will always be either -2pi or 2pi