You shouldn't feel guilty at all, doesn't make you less of a mummy, we're all different.
I went back to work part time when Holly was about 3 weeks old (working from home though, and only about 10 hours a week) and gradually built it up to now being full time (although Holly is only in nursery part time, I do quite a bit of work of an evening when she is asleep) (I'm self employed and needed to keep my business going so only actually had about 6 weeks full maternity leave).
Working keeps my brain stimulated, I love my business and I think (for me personally) it's important to show Holly the importance of hard work and financial independence.
Plus, she absolutely loves nursery - they are wonderful and adore her (as they do all of the children).
Do what works for you and what makes you happy - as much of a cliche as it is, happy mummy=happy baby.