The gap between feeds, the length of feeds, the weight gain.... normal normal normal!
I don't understand why you dread the next feed? Is it painful or is he fussy or what?
After he's fed for 5mins, what does he do... does he pull away and fuss or does he fall asleep or what? And is that 5 mins on ONE side? Do you offer the 2nd breast as well, do a nappy change or something at half time to give him a breather?
If you really want to give up, then do it... but be sure that it's definitely what you want because it's not easy to go back once you've switched. And don't quit on the basis that you aren't doing well at it because by the sounds of it you're doing great!