I typically stir my color in with a spatula, making sure it has been pre-mixed well with some of my liquid oils. Too much stick-blending will cause a thicker trace which is typically not desired when working with a swirl.
You can stick-blend though. I have seen many use the same blender, not rinsing or cleaning in any way. Usually, the small amount left on the blender won't cause a huge change in color. Just go from light colors to darker colors.
EDIT: Sorry, just realized you were talking about multiple batches, not just one batch with multiple colors. Could you wipe with paper towels between batches?
I would suggest using a tall-enough container for the SB to stand up in between uses that you can do a short burst in water to help clean off the bell & arm of the SB. Then tap it on a rag or paper towels to get off the excess moisture before using again in soap batter.
I use a plastic cup in which my SB fits perfectly to stand it up in between uses, but I don't use the water in-between method very often because I don't do multiple batches very often. I just remove the SBer arm and hand wash it when I'm doing more than one batch in a day. I think the most batches I've done in a day is probably 3, and usually I wash the arm between uses, although once or twice I may not have done. I don't like to let my SB to remain covered in soap batter for long because I had one actually loose the seal between the blade and the bell when I didn't clean it as timely as I do now. When I first started, I'd let the soap batter dry on the SB for a day or more before washing, and I believe the lye in the batter ate away at the adhesive and that's why the seal failed. It's never happened since I replaced that SB and always wash the SB immediately after making soap.
I work colors from light to dark. I pour off my batter at emulsion, add in my colors and quick zap with the SB going from light to dark. If I have to go back and SB a little longer I rinse off my SB, especially if I used AC as one color. When using mica I do not dilute them I just add the mica into my soap batter, which works best for thin batter.
If your batter is thicker it is best to premix the colors and stir them in. I lay my sb on a paper plate or grease resistant sandwich sheets, since my favorite SB does not have a removable bell. I soap next to my sink so I also have one side filled with soapy water so I can swish my SB to clean it