And your molds play a part as does the weather. Soap in thick silicone molds seem to take longer to harden in the mold than soap in a wooden mold with a very thin silicone liner or freezer paper lined. Gelled soap comes out faster than ungelled soap. Some people prefer ungelled soap, but if that is the case it's a longer wait to unmold. Sodium Lactate or dissolved salt can be added to the soap to facilitate faster hardening and earlier smoother unmolding.
FYI, some molds cannot be CPOP'd. If you ever make CP soap in plastic Milky Way molds, don't CPOP them or they may become warped and damaged. But I have no trouble CPOP'ing soap in a cardboard box lined with freezer paper. (CPOP = cold process oven process, which is what toxikon is referring to above.) I heat my oven to 150 F and turn it off when I put the covered mold inside. I check the temperature of the soap after a couple of hours and if it hasn't heated up to higher than the cooling down oven, I re-heat the oven again to 150 and turn it off again. The second heat-up usually hurries the gel phase along if it didn't heat up before. Some soaps require more help than others to heat up. Depends on the recipe. If your recipe is high in CO, it heats up and gels pretty easily without any additional help.
When I soap in the winter, my soap needs more help to gel as I keep my house on the cool side in the winter. In the summer I keep my house on the warm side, so I adapt my methods based on internal house temperatures as well as the recipes and molds, etc.
Another method to make your soap ready to come out of the silicone mold quicker and more smoothly is to put it in the freezer for a few minutes to a couple of hours (depends on the size of your molds.) This really helps soap come out of silicone molds easier, but you do want the soap to be ready for unmolding first. If you try it too early with ungelled soap, you can still dent it too easily. Once removed form the mold, let it rest and warm up to room temperature and determine if it's firm enough to handle. If it still seems on the soft side, let it sit out with air circulation and it should be ready to handle in a few hours or so.