Are you talking about C3D Labels...?
There's a long-standing problem where C3D Labels that are in XREFs will "freeze" at the modelspace scale set in the XREF. They size correctly in paperspace, even if they're in an XREF, but they might look VERY WRONG in modelspace.
The way it works is that you set the "print height" (aka the size you WANT to see in your final print) in the Label Style. This then gets multiplied by the Drawing Scale (in modelspace) or the Viewport Zoom Factor (in paperspace), so that the text in your final print is the size you set in your Label Style. Best thing is to ALWAYS use the scale setting in the status bar to change things... This will change both the Drawing Scale and the Annotative Scale if you are in modelspace, and will change both the viewport scale and the viewport Annotative Scale in a paperspace viewport. This keeps you C3D Labels and your Annotative elements in-sync.
To get back to the other issue, say you have some labels in your base drawing, and the Drawing Scale is at 1" = 50' in your base drawing. When you XREF that into your sheet drawing, your C3D labels will remain "frozen" at 1" = 50' in modelspace in your sheet drawing, no matter what you do. They'll size correctly in paperspace viewports in your sheet drawing, but in modelspace, they'll stay stuck at 1" = 50'.