You can not purge objects that are refernced in the drawing. Which means if a layer or text style is used in a block it can not be purged, if a layer is locked, or forzen in a viewport, it is refernced, and cannot be purged. If an object in the drawing is on any of the layers (in a block or not) it can not be purged because it is refernced.
Make sure all of your notes and symbols are on the correct layers, and there are no nested layers in any blocks. Also check to see if any of the layers are frozen, or locked, or forzen in a view port. Also make sure your text styles are the ones you want to use, and not refernced in any text strings to purge out misc. text styles.