Ashley used to be a boys name. So if you like it either way I don't see why you shouldn't consider it.
The only thing though is it would be a shame if Charlie becomes so trendy for girls it will no longer be a good boy name. It's very cool on a boy.