I'd say tell him it's not ok as made other plans and from now on you want things sorted out in advance - if he can't sticl to a simple arrangement then he can't just drop in and sweep her off when he feels like it! Remind him that his time with harriet is meant to be for the two of them - not him and his mate or his mum!!
Try not to sound petty (even though it's soooooo tempting!) but just make it clear that if he can't reach an agreement with you and stick to it then you'll have no choice but to go for a formal agreement and that means all sorts of restrictions and inteference from social workers and so on, you don't want to be difficult but you can't have someone just swan in whenever they feel like it and change arrangements whenever they feel like it, And at 5 months I would say she is a bit too young to be with him overnight if he's easily distracted by mates and the xbox!!