After reading the above post, it is an order of operation issue.
I was setting the value, then activating the viewport.
Same applies to my working methods, I do not ever set a UCS through a viewport.
I create them in Model space and save them with the view, that I later set current through the viewport.
So I haven't ever had the issue that is created by doing through a floating viewport.
So I guess, if one uses cad in a particular fashion, yes the UCSFOLLOWS variable will cause your viewports to zoom.
My suggestion; STOP doing it that way.