Ok, I thought maybe he was getting more filled up from the formula if he was having more of it than he is milk now, but if it's the same amount then obviously that's not it lol.
I switched ds over gradually and he was fine so maybe do half whole milk and half formula in his bottles for a few days until his tummy gets used to it. It shouldn't be too big of deal to his system though because formula is made from cow's milk, it's typically made to be easier to digest though. Also could be the temperature, were you still warming his bottles of formula or where they room temp.? A cold drink can sort of hit your stomach kind of hard, ya know? So maybe warm the milk a tiny bit too for now just to take the chill out of it.
