Here's an interesting article I found https://www.webmd.com/baby/news/20020408/pool-water-risky-during-pregnancy . A quick Google search will turn up many more articles saying there may be some risk.
Yes, a lot of people swim in chlorinated pools during pregnancy and their babies are fine, but that doesn't mean there is no risk at all.
For me, I would stay out of public pools during first trimester at least, since they are heavily chlorinated, and first tri is when you are most likely to develop birth defects. The time that you'll really, really crave the weightlessness of the water is in second and third tri anyway, when you start feeling really big and heavy.
I'm 37 weeks now, and swim every day, but we set up a small above-ground pool in our backyard and keep the chlorine levels really low; nothing like a public pool. You can buy inflatable above-ground pools for pretty cheap, and as long as they're at least a few feet deep, you can float and kick and do some good stretching. Having mine has made a huge difference throughout third tri. While most other people have back and hip problems by now, I am totally pain-free.