It’s a vicious cycle. When faced with a number problem you are gripped by anxiety or panic. Clearly, these feelings are no good so you go out of your way to avoid number problems as much as possible. This leads to your becoming less and less “good at math” and the feelings of anxiety increase. Round and round you go until your math anxiety has become a fact of life for you.