Yes, that's the problem.
I also let the silk soak in a tiny bit of distilled water, then add a tiny bit of the lye to that, and smash the silk with a hard spoon to dissolve while it's hot. I let that cool, pour it over my frozen milk, then add the rest of the lye as usual.
Don't breathe...