If he is not bothered then he is not bothered. See it as his loss and no reflection upon yourself at all. You have tired to include him, you tried to make things work... heck... you even tried when he split from his ex and came back to you. He might twist the situation to make himself feel better or make others think he tried and it was your fault but you both know... if he wanted to be involved no matter what obstacles you put in his way he would find a way.
If you see him when out then ignore him, continue with your night out as if you hadn't seen him [which I appreciate is easier said than done] and if he wants to chat then let him but only about the baby...nothing else.
Hope that helps and you are not a pushover, you just care about your baby...big difference.