You are completely justified in how you feel. Hell, my baby is 4 1/2 months old and I wouldn't leave him overnight with anyone, not even my parents (who he sees multiple times per week).
Can you have someone watch the baby for dinner, but have him stay overnight with you at the hotel? Its still a night away from home for you guys, it can still be romantic!