however...IF you Named the title block layers in such a fashion that the A size sheet layers could be ON while ALL other sheet size layer were OFF that would work.
You would of course need or want a unique layer name for each instance of the Xref attachment to have full controll of ALL Xref layers on each Layout Tab.