Want to move out of State

I recently had joint custody of my middle child granted to me and her father. We were never married. I have an older son that is not his and a new baby on the way. I would like to move to another state where I can own my own home, have a better education for my children, as well as get a better paying job and be "self Sufficient". When I brought this up to my daughter's father regaurding this issue, he stated I could move in 10 years. Does he have the right to decide that fate and whereabouts of myself and my other two children which are not his? The state I want to move back to is where my oldest child was born. He just recently wanted to have anything to do with my daughter since the courts told him so. I know he has rights but I have ALWAYS been there for my children and provided for them with out there father's. Can he make me stay? HELP