A few years ago, before my friends started having babies, we would talk about this quite a bit. I did wonder the same, did ttc successfully but then lost the baby to a fatal development issue (which they said was bad luck, not either of our doing).
So, short form, yes, I have felt that way. I think it's a lot of the unexpected and just plain not knowing. We can hope that everything is ok, and we can want everything to be ok, but the only true way to tell is if we do actually go through with it and try and successfully have that baby. How do I deal with it? (Now I wonder what loosing the baby has done to my system, etc, it never ends!). I just put faith in the system. The world functions because women have babies. The real clincher for me was when my partner told me if we did have a problem naturally, that he would do anything to have a family with me, including all options, surrogate, ivf, adoption. So, I know that we'll get there, it's just a matter of when is all.
I was so sure that it was going to take us forever to conceive because I was convinced that we would have issues that we started earlier than we would have other wise. We fell pregnant on our second cycle. I'm at the start again, wondering if that was my chance, if it will happen again, blah, blah, blah. But really, it did happen once, I'm grateful for that, I pray it'll happen again and everything will work out.