I can understand the point of view of the two posters above who disagree, and while if I were the FOB I'd probably want to be there too, that doesn't take away from the fact that it's not his body who will be doing the work, that will be exposed in a very personal and private way, so I think the choice lies solely with the OP.
I'd also be wary about letting him only be present for 'part of the time' because what if something happens and the OP wants him to leave at a certain point? I.e. maybe after the birth she needs stitches, wants skin-to-skin and time to BF, waiting for the placenta... those are personal moments that I wouldn't want someone I wasn't close to being a part of. Is the FOB really likely to respect the OP's wishes if she asks him to leave at this point? (OP, only you can know the answer to this). Just some things to take into consideration, but I can imagine that might make things more awkward or tense than necessary.
@ OP-have you invited him to any of the ultrasounds? Maybe that could be a compromise to let him feel involved in some of the more 'personal' aspects without being too intrusive?
It's a difficult situation