I have read that when using clay as a colorant, you need to break it down in liquid first or it will clump in your soap. Disperse the clay in a small amount of rubbing alcohol first to make a liquid, then add that to your soap. I have not tried clay personally, so you may want to do further reading.
The soaps are lovely as is, though!