*begin largely unscientific personal opinion*
My stance on most things of this matter have been this: Life is too short. That isn't to say turn a blind eye to everything. Be informed, know what your line is and where you draw it; however, you cant let every study turn you away from doing something you enjoy.
My reasoning is two fold
1.) Much of the data that people post around the internet is misrepresented. Things are misunderstood, facts taken out of context, and some things are flat out manipulated to seem one way or the other to serve someone else argument.
2.) If you really feel the need to worry about everything that's bad for our bodies just remember this, oxygen is slowly killing every human on this planet through Oxidative Stress.
Think of it this way, you are dispersing (usually) at most one teaspoon of mica/oxide/pigment per pound of soap, a wash off product. The lather in soap is there to lift up particles from the skin so they may be washed away with water. Many micas and oxides are used as makeup, put on or near mucous membranes and left there for hours on end, daily. It would seem to me, that if there were to be a worry about using micas or oxides that would be it.
Honestly i would be a lot more concerned about inhaling micas than having them absorb through my skin. So just don't huff your shiny polished mineral fragments and i'm sure you'll be fine.