No, don't worry! The way your breasts feel (and how much you pump) do not accurately reflect how much milk you are producing. The key is to feed on demand, and to offer both breasts at each feeding. Evening times are just a very very common time to have a fussy baby.