1.) Church history is descriptive, not prescriptive, in nature.
2.) Age does not prove truth. Does Buddhism and Hinduism have more truth than Christianity just because they are older religions?
3.) Even if we unanimously agreed to accept Papal authority, that would only eliminate doctrinal conflict in a question begging, tautological sense. That would still not reveal to us whether we should be in communion with the Roman Bishop (whether we are right or wrong in our decision making). A case for Roman Catholicism would still need to be made.
4.) Both the Roman Catholic and Eastern Orthodox churches make identical claims of being established directly by the Lord Jesus Christ, but maintain contradictory oral traditions and dispute each other's claims.
5.) The church fathers sometimes contradicted each other and themselves, which is the underlying reason for us not putting them on par with the authority of divine Scripture. These men, no matter how godly or theologically gifted, were not inspired by God. The writings of these people are not to be treated as a smokescreen against the plain reading of Scripture. Divine revelation exists independently of the writings of so-called church fathers. Heresy was present among the people of God even during the first century (Acts 20:28-32; 1 John 4:1-4). So it is best to stick with Scripture as our guide in the development of doctrine.
6.) Roman Catholics cannot know with certainty who exactly constitutes the church fathers, which early Christian writings are to be deemed authoritative, and that all of the father's teachings are correct without resorting to circular reasoning. For them, matters are thus because the Church of Rome declared them to be thus. That is circular reasoning.
7.) We do not have every document written by each church father on every subject. Neither were we present in the early church to take surveys of what everybody believed. This demonstrates the Roman Catholic claim of unanimous consensus in church history to be vacuous.