I've always followed the 6 weeks rule for cold infusion. I don't let any leaves stay in longer as they could start to get funky and spoil, but leaves do not really add much color, other than brown.
Spices I just do the same with. Don't know if leaving, say, annatto seed in longer will make a difference. Certainly using a lot or a little does make a difference.
Calendula petals reach yellow and don't get any further after 6 weeks. I have unintentionally let that one sit longer. Actually, by 4 weeks they are not making the oil any more yellow then they already have, in my observation.
By the time 6 weeks is up, I am ready to strain them and use anyway, 'cause I am tired of shaking them daily, I am lazy, lol.