I have not heard of using an infused oil added at the end as both the superfat and the colorant, although it seems like it would work provided your colorant is strong enough to show at 5%.
Earlier in the thread Debbism posted a list of what percentage of your recipe's oil needs to be your colored oil. There were 3 oils that should color nicely at 10%. If used at 5% you should have some color remaining after cure. For pretty much all the other colorants, you need to have much more of your batch's oils be the infused oil. So I'm not sure what resource dictates that 5% of a colored oil will color the soap, unless the person was talking about micas or lab colors. Or was just wrong!
One safety/quality point: If you are keeping the superfat out of the entire batch of soap, pouring out half the batch, then adding the superfat for ALL the soap to only HALF the soap...well, that would make half the bar harsh and half the bar super-super-fat. If your scale is inaccurate, half the bar will actually be lye-heavy. I don't recommend any of that.
Or, as Bodhi surmised, if you are giving the uncolored half its uncolored superfat, there is no safety/quality concern. I just wanted to bring this up in case you're doing it the other way.
You could melt together all your batch's oil, including your superfat, then put part of the batch in the crockpot on low (or a double-boiler, or a pot on very very low) and infuse lots of oil at once. When you infused the oils, did the tea bag keep all the material out of the oil? If specks do fall from the tea bags (and you don't want any specks) you could find something to strain the oil -- cheesecloth, maybe?
Lastly, some botanicals are very scratchy, like paprika or anything in big chunks, but some are soft, like cocoa, rhubarb root powder, alkanet root, spirulina (goes brown!) and even parsley flakes (which stay green for awhile).