The breast is never empty because it is always making milk. It doesn't have any storage areas for milk, which is why at the beginning it is so uncomfortable and engorged because your body is making milk that can't go anywhere but just get backed up in the ducts. Once supply regulates boobs rarely get engorged because the breasts know how to make milk quickly on demand. In terms of when to swap, I'd say when baby is unsettled and rooting but refusing to latch to that side you should swap.