I have used actual ground spices in my soap - allspice, ginger, cinnamon, clove, nutmeg. You do get a bit of a scent from those. But I don't think that it's possible for us to infuse enough of the aromatic compounds in water or oil to survive the lye monster. Smell the strength of an essential oil and then smell the strength of an infusion.
Second user of the powder. I did 1 tsp per pound of soap (before I knew it was per pound of oils to measure against). There was some scent originally but it's very very light two months later. I love the speckled brown look it gives though! Have not infused though.