It's one of the teas that could *possibly* cause miscarriage due to being estrogenic. You'd likely have to drink a lot of it (multiple cups a day) to have any effect. Most reas that are considered "possibly unsafe because they haven't been well enough studied" are acceptable in moderation (one cup a day or less).
Even some herbs that do have proven abortifacient effects in large doses would be safe to consume as a tea in moderation (though obviously not something many people would want to chance)
A tea is literally the weakest herbal preparation. To actually see a benefit from herbs (for example the red raspberry tea often consumed at the end of pregnancy) it really needs to be as an infusion (steeped for 8 hours). 3-5 minutes of steeping has no potency.
Now I'm not saying go ahead and finish the fennel tea. That's one I really am not very familiar with and just looked up quickly on some of my trusted herbal sources, but I do hope it will ease your mind about what you have consumed so far already.
Also, chamomile, peppermint, red raspberry, passionflower and many more all have the same classification. *Possibly* unsafe in large doses, but fine to consume in moderation (1 cup a day or less)