Why can I do to stop him controlling my visitations?

I am seperated from my ex, in fact trying to get the money for a divorce. I am paying child support even during the months that I have the child. Was told it has to be that way so the child can have a guarantee home when they get back. He is no longer working. He quit about a month or so ago. While the kid was visiting he was evicted from his home and that it was a mess everwhere when they moved. They get medical and food help from the state even though I had the child for 2 months. No meds was sent due to not getting a recheckup appt. I was told it was the dr's decision of none for the summer. But found out different at a later time. I have to talk to his gf about everything. He won't. She calls me with messages that I need to call her to talk about the kid. She texts me about when I am to have the visits and when to have them back. I know that she has even said some things about the dad to my child. I can not buy school items for them because I will get cheap things. I was told that he had gotten more tattoos though he has no income. I know I should had did things different like not leaving the child to get a place setup to live. But I believed what I was told at the time. I have been told that because there is no divorce I have right to keep the child living with me. But I don't want to do anything illegal. I don't know what I can do. But I think that the child needs more than what he/she is getting attention or other ways wise.