My doctor has always told me to get back to it about 6 weeks postpartum, and I've had complication-free vaginal deliveries.
You can probably do some lighter stuff before then (short walks etc), provided you feel up to it. But if any type of exercise feels weird, especially in your abdomen, causes bleeding, etc - then you need to back off.
I'm not sure how/if it changes for c-section deliveries.