Assume you are using System.Windows.Forms.Form as your modeless dialog box. If you show the modeless dialog box correctly (i.e. using Application.ShowModelessDialog(), not Form.Show(), as Autodesk's development document recommended), then setting form's "StartPosition" will have no effect on where the form is displayed: AutoCAD remembers where the last time the form was positioned. Also, in most cases, the modeless form usually is a static/Shared instance during an AutoCAD session (i.e. only a single instance of the form is created, and its visibility is set to true or false, based on needs. If each time when user close the modeless form and you dispose it and create new instance, AutoCAD open the form at the last position with a small offset.
To force a preferable StartPosition while still using Application.ShowModeless() to open a form, you can handle Form.Shown event and write code to position the form. You can use System.Windows.Forms.Screen.Primary/AllScreens to determine the primary screen, or the desired screen you want to form to be, you can also user Application.MainWindo to find AutoCAD window's size/center for your form.
Just remember, Form.Shown only fires once when the form is first shown. If you only have a singleton modeless form and toggle it on and off, but wish it to be a certain location when it is turned on regardless where it was turned off, then you need to handle Form.VisibleChanged event to manipulate its position.
