I also had a Close Parent, which was great when Adam was tiny, but I never used it for longer stretches than a 20 minute walk to the shops. That was mostly because I ached until Adam was 6 months old. By the time I stopped aching, he was really too big to be comfortable facing in, and although you can carry them facing out and that was comfy, it's not good for their hips.
Up till 6 months, I used the sling maybe once a fortnight, more in the earlier days, but like I said, never for longer trips. I don't blame the sling but I do blame my aching bones. I almost was going to give up on the idea as I thought he was getting too heav but decided to get a Connecta and persevere, simply because I had a few bad experiences with trains and steps and the pushchair and I really really wanted to be able to go out for a while and not have to plan my route around step-free access.
I am SO glad I bought the Connecta. Adam is now over 21lb (he was last weighed 3 weeks ago) and of course it feels like I'm carrying a 21lb child but it doesn't pull at all, the weight is distributed nicely over my shoulders. I can go out for 3-4 hours and feel fine. I am also not that big, 5ft 4in, and not fit (though fitter, I am sure, since having him!). I love carrying him, it's so quick to pop up to the shops and back compared to getting the pushchair out (we live up 2 flights of stairs), and he seems to like it as well. he will happily fall asleep after a while in it.
I don't think it's ever too late to start, unless you honestly cannot bear the weight, but with the right sling you won't feel uncomfortable.