I've done it plenty of times, once by packing the car out of the garage and clipping the trim piece, and another time by parking on a busy street while another car clipped my driver side mirror. The actual mirror piece pops right out. You might want to move it with the mirror controls so it is all the way up and in or down and out to be able to get some fingers behind it and pull. It just had a few plastic clips behind it to hold it to the mirror motor mechanism. The driver's side also has a wire presumably for a defroster 9this is on a gen1, I presume the gen2 is similar.
So why are you trying to remove the mirror caps?