Talk to a Lawyer where you live.
Can A Father Move House And Take His Children Without The Mothers Consent?
If the father has full custody of the children he can do whatever he wants with the children and the mother has no say. If there is joint custody then he can't.
I think he can if he has cause, like you abuse the kids or get high on drugs everyday. Otherwise no he can't do that. You have legal rights as well as he does.