I can relate - I started using CP soap about 6 months ago and experienced what we're now affectionately calling the Plumbing Fiasco of 2011 in December. It was a combination of problems for me -
Is your house or plumbing older? Mine is from the '40s with some original plumbing, so we're talking cast iron. Cast iron that has had 70+ years for crud to accumulate, which makes the interior surface of the pipes rough. The oils in CP soap (instead of detergents in many commercial soaps) really love to cling to the rough crud on the inside of old pipes, then it just becomes a cascade of more and more accumulation.
After the plumbers were called out, I became more proactive - I use a fine mesh screen over my drain, which collects a surprising amount of soapy buildup. Additionally (I read this online somewhere), I squirt about 1/4 cup of the original blue Dawn down the drain every week or so, the logic being that it helps strip away some of that soapy/oily buildup.
Worth it for good soap? Totally.