This happened to me a couple of times with goats milk. It turns out I let the temperature of the lye and oil get to cool (85 degrees). Now I soap at 90 and never have this problem. Then I put in the freezer over night so it doesn't gel, remove from mold and cut 24 hours later.
I do it all the time. I only use clays and oil infusions for my color. I read that the clays help hold the scent, so that is what I do for every batch. I never had a problem!