"But because of immoralities, each man is to have his own wife, and each woman is to have her own husband. The husband must fulfill his duty to his wife, and likewise also the wife to her husband. The wife does not have authority over her own body, but the husband does; and likewise also the husband does not have authority over his own body, but the wife does. Stop depriving one another, except by agreement for a time, so that you may devote yourselves to prayer, and come together again so that Satan will not tempt you because of your lack of self-control." (1 Corinthians 7:2-5)
If Mary refused to have marital relations with her husband Joseph, then she would have been guilty of sin. In order to remain consistent, a Catholic would either have to reject the notion of her remaining a virgin for her entire life or her sinlessness. Both cannot be true at the same time.