I say they wouldn't know what a friend is if that's how they treat them.
The way the child is and it's future are up to the parents that bring him or her up. I can't see why anyone would have the right to judge someone based on what they think their motives are, rather than actually taking the time to find out some facts. On top of that, like you said, it's none of their business anyway.
That's like my friends telling me they don't think I should marry my husband and have a child with him. It has about as much to do with them as my current consistency of cervical mucus.
I think that it's wonderful you have taken the time to think about what is best for all of you and to pick someone that fits with you and your plans for the future.
Bottom line, none of their business. If this man is bringing hope to people that otherwise have none, then good on him!