The xref may be polluted with one of the three main pollutants these days.... Scale entries, Regapps, and DGN linetype bloat.
We have a nice little reactor that reports the time it takes to load xrefs so over time, I've learned what to expect in terms of time it takes to load them.
When a polluted one comes along, the increase of time is usually obvious.