I've never heard of using annatto for m&p but I suppose you can try and make it work. Do you ever add any extra oils to your base? If so, infuse the oil with the seeds then add the max amount of oil to your base.
You might try adding some seeds to a small amount of base and keep it melted at the lowest temp possible, see if the color will infuse the base directly.