Mels love, its down to you and your ex on how much time your little one spends with his dad.
Makes no odds if he lives with you or not, there is no law saying he cant stay at his dad’s on weekends.
But this is more to do with your current B/F isnt it?
I think you both need to have a wee sit down and chat about it and let him know that you want to stay to the arrangement that has been in place all the last 2+ years.