You are definitely not alone. Last pregnancy I had a bit of anxiety and depression. The sad feelings went away after birth, but the anxiety got kind of extreme and I ended up being treated for postpartum anxiety. With this pregnancy, my anxiety has already gone through the roof and I'm worried it'll only get worse.
Do you have anyone you can talk to? I'm going to talk to my midwife about my anxiety at my first prenatal appointment. Like you, it also took us a long time to conceive (I ended up needing Clomid) and I think those difficulties made this more of an emotional roller coaster. At times, I'm convinced I don't deserve this baby and that something will happen to the pregnancy.
Anyway, I'd love to be a support buddy if you need someone. I know we can get through this!