Briss-from my own experience, I have to tell you that I was DEVASTATED when I was told that I would need IVF. Like you, I felt that I just needed to carry on tinkering with different meds, for both me and OH, and we would get lucky. I don't think anybody EVER starts out thinking that they'll need IVF, but, a lot of the time, the decision is thrust upon you, when other things fail.
If I had my time again, I would go straight to IVF. I started trying when I was 36 and so already past the 35 'Oh my gosh, will it ever happen' fertility decline. It took two years to get one BFP, which miscarried. Foolishly I again thought that I would get another chance, but this time at least I did press for IUI, which was aborted. Long story.
Only you know when you are ready, but you have to make your mind OPEN to assisted conception, and that's what I refused to do for a long time. And by doing that, I wasted precious years.
I know, for a fact, that fertility clinics are frustrated that women are presenting themselves to them with years of futile TTC under their belts, when the stark reality is they could have far better hopes of success when the woman is younger.
I'm conscious that I sound as if I'm TELLING you what to do, but if I could have my life over again, I would not pussy foot around the issue. We are over 35, so we are already in a reduced state of fertility so we don't have the odds in our favour anyway. Yes, you will ALWAYS hear stories from those who will fall pregnant at 45 in their first month of trying, but these people are the exception, not the rule. So, I'd just get on with it!
Sorry to just spell it out-what you want (a baby) is just too precious to waste time over and then live with regret.
Good luck!