As readyformore said, if stress caused infertility we wouldn't have pregnancies from rape, abusive relationships in extreme poverty and famine etc yet it happens all the same.
My wee gran had 11 kids, when she had her first she lived in a room and kitchen. I can imagine by the time she had 3 & 4 kids she would have been pretty stressed year she went into have more and more.
And just as an extra point about ltttc, she didn't breastfeed or use contraception but it took approx 2 years to get pregnant again between each child. Sometimes we worry so much about it bit our bodies are just doing their own thing. You could never say she was infertile, but if one of us were to take two years to fall pregnant we would be worrying about infertility.
It's one of the few things that comforts me these days, keeps me sane.