No, I don't think it's just you. But I don't know of a good solution to prevent streaks and smudges short of packaging the pristine bars of soap in shrink wrap. Maybe others will see this an share their tips now that I've "bumped" your thread.
I'm not a great fan of dark colored soap to be honest. When I made a mostly black soap using black iron oxide and/or activated carbon, I found the lather was gray and the soap discolored light colored washcloths.
I still like to use black to make a dark accent color in otherwise lighter-colored designs, though.