If you are certain that he's the father, and if you intend to hold him accountable (as you probably should) then you need to allow him limited access -- particularly if you want a friendly relationship with him.
He will then not be in a position to tell a judge that he wanted access to his child, but you denied him.
He will then not be in a position to tell a judge that he wanted access to his child, but you denied him.
You haven't given us much information but since it's no small thing to raise a baby on your own it would be best to seek legal advice so that you're covered in all eventualities.