I think that you are better off without him, especially for the time being. I mean, its normal to freak out over being a parent, but the way he is dealing with it is rubbish.
I know I'm simplifying, its always more diffucult when its you, but you need to not be stressed and constantly wondering what the problem is.
And your mom sounds like my mom. Almost parallel. But at least she is trying.
And they aren't all bad, men. It took me 28 years, but I found one.