HE doesn't have any rights, but your LO has rights to see FOB :/. But I totally agree if he's only going to be there when he feels like it it's better to not be there at all. If he was to take you to court they'd want to know what efforts he's made in the past, so make sure you take notes now just in case of any texts/calls/emails he might send asking to see LO. If that's none it's going to work against him. They wouldn't take any consideration to the fact he hasn't and doesn't pay a penny though. They will take his drugs into consideration, and you could use it to push for supervised until you're satisfied that he's clean and capable.
If he were to take it to court though and he was to miss them and not turn up etc. and was to mess around, then the courts don't take it lightly and he could be fined and have the contact order taken away in which he has no rights to see LO at all, but it has to be a regular/constant thing that he's not turning up.