It is very important to make sure you keep your cats and dogs reguarly wormed because even if you can't see them they can crawl out and up the fur and can be found as far up as the middle of the pet. Some parasites you can't even see like the Toxocara (causes toxocariasis) & Toxoplasma (causes toxoplasmosis). Really though it's just about good hygiene. Cats and dogs like to roll around in the dirt, with who knows what in it, then young kids like to stroke the pets and being young kids put their hands in their mouths but you can't be afraid of every little bit of dirt.
The other thing like someone else mentioned is to make sure you never leave your pet and child in the same room unsupervised, no matter how much you trust your pet because animals are unpredicatble. Make sure cats can't get into the room your baby is sleeping in.