Critics can't propose multiple layers of development behind the infancy narratives, with earlier sources making claims that Matthew and Luke repeated, yet turn around and object in other contexts that Matthew and Luke are the only early sources who made the claims in question. As Charles Quarles notes regarding the notion that multiple sources behind Matthew's gospel affirmed the virginal conception:
"That allusion or affirmation of the virginal conception appears in multiple pre-Matthew sources should make one pause before dismissing it too lightly." (in Robert Stewart and Gary Habermas, edd., Memories Of Jesus [Nashville, Tennessee: B&H Publishing Group, 2010], approximate Kindle location 4168)
Critics can't have it both ways. They can't argue, on the one hand, that Matthew and Luke were alone in claiming a virgin birth, yet argue, on the other hand, that they (Matthew and Luke) received the claim from earlier sources.
Furthermore, 1 Timothy 5:18 seems to refer to Luke's gospel as scripture. The implication is that the author of 1 Timothy (Paul, I would argue) is indirectly affirming the virgin birth.
When we get to the patristic era, the virgin birth tradition is affirmed early and widely. Ignatius, writing as bishop of a Pauline and Petrine church, affirms the doctrine while writing to other apostolic churches (Smyrna, a Johannine church, and Ephesus, which was both Johannine and Pauline). Ignatius' comments to the Smyrnaean church imply that the Smyrnaean bishop, Polycarp, affirmed the virgin birth. Polycarp was a disciple of the apostle John. Celsus, a second-century critic of Christianity, thought the virgin birth tradition was so early and so widespread that he attributed the virgin birth claim to Jesus himself.
Since we have good evidence that Mary was pregnant with Jesus prior to her marriage to Joseph, and the early Christians acknowledged that premarital timing of the pregnancy, we should ask why that suspicious timing wasn't more of an issue among the early Christian sources. If Paul, for example, knew that Mary was pregnant prior to marriage, yet didn't believe in a virginal conception, then why is there no indication that he or his opponents were addressing such a scandalous situation? If the virgin birth tradition was early and widespread, then that would better explain why Paul and the other early Christians seem to have believed in a premarital pregnancy, yet thought so highly of Jesus and didn't think there was any scandal to address like there would be without a virgin birth.