I've used paprika fairly frequently, and it does have little scratchy bits. When I used it at 1 tsp ppo to colour the entire batch (as opposed to swirling), I found the soap to be annoyingly scratchy. But then, I like soap to be either dead smooth, or nubby like oatmeal. Scratchy, I don't like. (I also find activated charcoal to be a bit scratchy, so I may just be a sensitive, irritable person. Actually, I guess mr surf girl would say "oversensitive" and "irritable" describe me to a T). I only notice a little red pepper smell from paprika if I use it in unscented soap, but it's not a strong smell.
Cocoa powder does not seem scratchy to me, nor does yellow dock. Actually, come to think of it, I notice a bit of drag with the cocoa powder if I use it for a fine dark line between soap layers. Stevia powder can be a bit scratchy. I haven't used my soaps coloured with tumeric, curry powder, elderberry, or rosehip. I should give them a go and let you know.
(Oh, and the colorant thread is listed in the photo gallery under something like "colorant experiment").