What I want is modify the block nested entities. to be bycolor or bylayer
In that case you'll need to iterate over the block component entities found within the block definition; if you still want the user to select the blocks that are to be modified, I would suggest iterating over the selection and, if the block hasn't previously been encountered, process the block definition and add the block name to a list of processed blocks.
I think there are quite a few examples of this to be found on the forums - for example,
this will change the colour of block entities (and nested block entities).