Yes, a reaction does occur. Aluminum and sodium hydroxide react to make sodium aluminate and hydrogen gas. Hydrogen gas is extremely flammable and potentially explosive. Sodium aluminate can be a skin irritant. The risk from the hydrogen gas is long gone, but the sodium aluminate is a concern.
There's no way to know for certain if the amount of sodium aluminate in your soap is enough to possibly cause irritation or if it's not a problem. It might be perfectly fine ... but I don't KNOW that to be true with the facts I have on hand. Perhaps others have better information to offer that would enlighten us all.
It's one thing for you to take a risk with your own body. It's another matter entirely for you to knowingly expose others to that same risk. And it creates a thorny ethical problem if you do not disclose the risk to those people.
In my opinion, I'd err on the safe side.