As I understand it, the Gentleman could always come and go to the human realms as he wanted. He just... didn't want to, possibly for the same reason as English (and, presumably, human in general) magic had been functionally gone. Something like everyone is far too dull and ugly to bother with now. When Norrell summoned him, he gained a renewed interest and started to meddle again.

I don't know for sure, of course, but that's how I read it.