Refresher course on blocks and xrefs:
A. A BLOCK is a "container" of entites.
B. An Xref is a BLOCK that has a path to its home with a few enhancements like: personalized layer names.
Now, every object in autocad needs to be regenerated when autocad loads. Xrefs take longer to regenerate, because not only does it have to regenerate, but for each instance, it has to check and make sure it's pulling in the correct graphical and non-graphical data. The more blocks, as stated above, the longer it'll take to open any drawing. Now, you probably don't use images much, if at all, but this is where my company slows to horse to a snails pace. The designers attach images and set them where they belong, then they copy all the images off to one side so they can set a viewport to them. Of course, nobody seems to realize that each viewport increases regen time, since they haven't learned the importance of freezing layers in viewports or freezing at all for that matter. They turn layers off instead of freezing them. Oops, sorry for the vent- Where was I? Images - Every time you pan or zoom an image into view it has to regen. My suggestion would be to learn all you can about the importance of viewports over multiple block instances and images and layer freezing vs. off.