Having a credit card with no balance (ie paid in full every month) is a much better way to build/maintain credit rating than a car loan. Car loans are terrible imo. We have never had a car loan ever, and our credit scores are just a hair below 900. (Though honestly most of that is due to having been out of the country for 10 years with a 10,000 line of credit open the whole time). We paid off all our debts in full before leaving but didn't realise the line of credit would expire lol). We did get in a bad amount of debt prior to that and vowed never again, had no credit card or car payments at all in the other country we lived in for a decade and would have been in an excellent position with our credit rating to buy a house if we had wanted despite not having any debt.
Currently we do have a credit card again, which we use to collect points on our every day purchases and pay off in full every month.
Carrying too much debt, even if you never miss a payment, can hurt your credit rating far more than it helps it.
So yeah if having a $400+ car payment is in order for you to have a credit rating to some day but a house, and if that same car payment is the primary factor to you needing to go back to work after having a baby, I would definitely be ditching it. A credit card in your name, even if you make one small purchase on it and pay it off right away, is going to help your credit so much better without causing any financial burden.