I have read a few stories about parents changing the names of their children, if they just 'don't suit' etc. One mum called her daughter Sophie, it was between Sophie and Isadora. When she was two they found that Sophie seemed to be a really popular name and it put her and her OH off it, so they gradually changed her name: From Sophie, to Sophiedora to Isadora of some sorts, making sure she was happy all the time. By the time she was three she knew no different, most children don't develope long-term memories before they are 4 anyway. It sounds sorta cruel, but I don't think it is.
There was another case of a wee two year old who was adopted as a baby but eventually got given back to her birth mother and her husband (the man had not given up his parental rights, it was a long court battle lasting the full two years) when they received their daughter back they changed her name back to what it had been at birth and now she's in her 20s and has no issues.
I know you said you didn't want to do anything as dramatic as that, but I what I'm trying to say I guess, is that it's ok. If you feel strongly enough about it and you and your DH think the same - it's worth considering and happens more often than you think! I'd give it a bit longer I think, and see if the name clicks (as I said I love Daisy but Phoebe was also my second choice) and if it keeps nibbling at you then it might be better to change it

Did you ever have this problem with your other kids? x