Adoption

This is a complicated case Im married with three children my first is from a previous relationship and his father has had little to do with him since he was born. We lived in Arizona up the street in fact from his father and he was suppose to take him every other weekend however he canceled more than half of the weekends he was going to take him because he would want to go to a concert or a party and when he did take him for the weekend he usually got a babysitter so he could go out and drink. He paid child support maybe a total of 8 months at 200 a month but alot of the time he would tell me he had bills to pay and couldnt afford it (he paid weekly 50$)
When my husband and I moved to new york he informed me he wasnt going to pay child support for a child he couldnt see when he wanted to. He told me i should stay in Arizona since my husband was then in the military and would be gone most of the time. We were in new york for approx. two years and he called three times while we were there, he did pay for a trip for my son to stay with my mother in arizona so he could see him my son was there for three months when he came home he had no clothes that fit him he had lost weight and didnt want to sleep in his bed anymore. It took us months to get him eating right and sleeping in his own room again. While he was in Arizona he would stay with his father at night after he got off work around 10 pm I called his house a few times and found that the people in charge of "babysitting" were in fact sleeping and his father would be gone and he would be awake and some of the time they didnt even know he was there or didnt know where he was, at the time my son was 2 1/2 yrs old. We recently moved to Kansas and his father hasnt seen him in almost a year, when i called him to let him know our new phone number and address he had moved a month prior and didnt tell me. I did find him to give him our new address and phone number but he doesnt call unless i tell him to i have stopped telling him he needs to keep in contact mostly because i want to know if he will do it on his own.. for a few months we didnt hear anything until i told my mother (who lives with us) that my husband and i were looking into adoption since my husband has been the only dad he has known since he was 8 wks old. She contacted him told him what was going on and he called here and told me he wants my son to go out and visit again at first i was ok with it and told him because of the problems last time he would need to put him in daycare he told me he didnt have the money and my son wouldnt even have his own bed. I told him he wouldnt be going out until i saw proof that he would have his own room at least his own bed and was set up to be in daycare. My mother has been telling him she will get me to let him go I am not changing my mind and my mother isnt helping by getting in the middle of things i dont know what to do anymore my husband has been his sole provider since we were married his father has never kept insurance on him or even taken an active role in his life unless it suits him. I dont know what to do. He cant afford to pay child support because he spends his money on drugs and beer he refuses to quit and when i told him that he needed to while my son was there his solution was to do it at work and not at home. I told him that didnt work he had to quit my mother told me its not a big deal I think it is. we have never been to court because we wanted things to stay civil but the older my son gets the more he notices and i dont want him exposed to the things his father does. I have given him four years to get his stuff together and he hasnt really made an attempt.
I need to know what i need to prove he is unfit and what i can do so my husband can adopt my son before this goes any further.
Will the courts give his dad more time....?