My friends and I all have very "old-lady" hobbies, both knits and I make soap so we always joke about it and the term "old lady" became an endearing term.
I say, don't let it get to you. The soapmaking hobby is also seeing an increase of younger hobbyists ! Btw, I'm 21 in Sept !