Im pregnant, my boyfriend asked if his friend could stay w/ us for a month... Is it wrong to say no?

Im 20 years old, my boyfriend is 23 years old and I am 5 months pregnant. We just moved into our own house about three weeks ago. Before that I lived with my mom and he lived with two room mates who are very good friends of his. But let me say, I hate those two men. They are the most disgusting low lifes ever. Both have been to prison, one is 28 and the other is 32. One of them works part time with my boyfriend who got him a job, and the other doesnt work at all- he scopes out older woman on the internet and manipulates them into buying him things and paying his rent (true story). Both of their lives revolve around partying and , drinking and women. thats IT. Me and my boyfriend have a lot of trust issues, we have BOTh betrayed eachother numerous times in numerous ways. But we are in love and we are trying to fix things. Because of things that have happened in the past, I told him when I first got pregnant that he was not aloud to live with me and be with me if he was going to continue his ways of drinking and going to bars, and that if he wanted to be with me he had to stop. (I never threatened and never would take his child away). However if he wanted to continue that lifestyle, I told him we could split up and have joint custody of the baby when he's born. ANYWAYS he chose to stop the partying, and now we live together. But not he wants one of the former room mate to live with us the month of March, while he is waiting to move in with other mutual friends. I was in a good mood when he asked me, and I told him yes- so long as he doesn't have girls over at the house and so long as my boyfriend still doesn't go out to bars and come home late and start pulling bullshit like before. but now im stressing out hard, I hate this guy. He's disgusting, they work together- I just know that the second he moves in my boyfriend will start coming home late and coming home drunk and making excuses to stay out... well maybe I don't know, but Im not willing to take the risk. Before moving out together I told him that neither his or my friends could ever live with us, that we are doing this to become a family and thats the end of it. But what do I do? He's going to be really upset if I change my mind.. But I do not want this man in my house. Not to mention, I don't want to be stressing all the time about when he's going to be coming home from work with his friend and stressing my baby. I am already a very pessimistic, high anxiety paranoid person with a lot of trust issues. so anyways, here I am at work dreading the first of March to have this beats come live in my house. HI feel bad that he has no where to go (he will have to move back to LA with his mom, we are in San Diego), but then again he's 28 damn years old and I'm not a babysitter... Any suggestions?