Maybe I'm a weirdo, but I honestly have never noticed! I usually do 4 batches per soaping session and the first thing I do is measure my distilled water into glass jars for each batch and add my silk. After I arrange all my soapy stuff, I dump the water for the first two batches into my lye pitchers and add my (un-dissolved) sugar first and then my lye. I stir for a few seconds until I feel the heat and then set them aside to cool down (I usually soap around 110-120 degrees). I give them another stir after 10 minutes or so to help dissolve the silk, and I've never noticed the lye water being cloudy...but adding sugar does turn it yellow!
After I make a batch of soap, I repeat the process so I don't have to wait for my lye water to cool down after the first batch. This has worked well for me in order to make the best use of my soaping time!