Can you set up a joint account and both add a fair amount each month (by direct debit)? Then you can use that money to buy joint things such as baby things, meals out etc. And use your own accounts for personal things, like his days out.
To be honest my OH and I are very fair and organised with our money and it would really irritate me if I was in your situation. I would be feeling just like you.