No BTDT, but I had almost no contact with my extended family for various reasons, and didn't have any problems for it. Anyone who tells you that babies need grandparents/aunts/uncles/cousins is full of BS. Now that I'm an adult, there's a few people in my extended family that I've decided to get in contact with, but I'm not in the least bit bothered having not known them when I was a kid. My parents had really good reasons for not wanting to be close to their family members or wanting me to be close to them when I was a kid.
If you don't think FOB's family is a good influence or if they just can't be bothered, don't feel any pressure to include them in your or your child's life. If FOB is still involved, he can do it. It's not your responsibility.