I struggled with having those horrible thought a lot with my last pregnancy. I even had a dream that my baby had cystic fibrosis.
I'm an OB nurse, so it's thrown in my face on a regular basis. It's not like I can turn a blind eye. I just kept telling my husband to reassure me that the baby was ok. It helped.
Worring is just part of parenting. First you get pregnant and then you're worried you'll loose the baby. Then the baby grows and you're worried it will have a birth defect. Then the baby is born and you worry about SIDS. Then they get older and you worry that they'll get kidnapped or molested, or any of those other crazy things that only moms worry about. It's a normal part of parenting.