1.) Church history is descriptive, not prescriptive, in nature.
2.) Age does not prove truth. Does Buddhism and Hinduism have more truth than Christianity just because they are older religions?
3.) Even if we unanimously agreed to accept Papal authority, that would only eliminate doctrinal conflict in a question begging, tautological sense. That would still not reveal to us whether we should be in communion with the Roman Bishop (i.e. whether we are right or wrong in our decision making). It still does not build a case for Roman Catholicism.
4.) Both the Roman Catholic and Eastern Orthodox churches make identical claims of being established directly by the Lord Jesus Christ, but maintain contradictory oral traditions and dispute each other's claims.
5.) The church fathers sometimes contradicted each other and themselves, which is the underlying reason for us not putting them on par with the authority of divine Scripture. These men, no matter how theologically gifted, were not inspired by God. The writings of these people are not to be treated as a smokescreen against the plain reading of Scripture. Divine revelation exists independently of the writings of so-called church fathers.
6.) Roman Catholics cannot know with certainty who exactly constitutes the church fathers, which early Christian writings are to be deemed authoritative, and that all of the father's teachings are correct without resorting to circular reasoning. For them, matters are thus because the Church of Rome declared them to be thus.
7.) If we can freely interpret the volumes of writings produced by the church fathers, the official Roman Catholic catechism, or encyclicals, then how come we cannot also make interpretations of Scripture? Only the cults demand blind submission to an earthly charismatic leader. If Roman Catholic apologists want to consistently apply their method of validating Roman Catholicism via history, then why not take things a step further? That means we start with the inspired writings of the New Testament.
8.) We do not have every document written by each particular church father on every subject. Neither were we present in the early church to take surveys of what everybody believed. This demonstrates the Roman Catholic claim of unanimous consensus in church history to be vacuous.