Assuming some things (as Glenn R mentioned)
Try this:
// definition
[DllImport("user32.dll")]
static extern IntPtr SetFocus(IntPtr hWnd);
...
...
// use before invoking the editor
SetFocus(Autodesk.AutoCAD.ApplicationServices.Application.MainWindow.Handle);
Even though the palette is modeless, it still has focus and your code is asking for a selection it cannot complete.
This worked. Thank you very much, Draftek.
I tried setting KeepFocus to False and using the SwitchLayouts routine just to see if those also worked, but they don't appear to be applicable to my problem. I think KeepFocus = False would work if it was my palette that was taking focus, but I'm pretty sure it wasn't the palette...although I still have no idea what actually was taking focus. I guess SwitchLayouts didn't do it because model space was always the selected layout, I just wasn't "in" it for some reason. Although these didn't solve my particular problem, I wasn't aware of these things and I'm sure I will use them in the future, so thanks.
BTW, Glenn, I just figured that this was more of a common issue, like using transactions or something, and didn't really require a lot of specifics. Mighta just gotten lucky though and it's not like it hurts to throw some of that info in there, so I'll try to remember in the future. Maybe I can just make it my signature!
Thanks again everyone. This forum has been a life saver for me.