1.) Church history offers descriptions of what people have done in the past. Its purpose is not to prescribe what our beliefs ought to be. The only kind of tradition that has been reliably preserved for us is that recorded in the New Testament.
2.) Longevity does not prove truth. Does Buddhism and Hinduism contain more truth than Christianity just because they are older religions? The fact that an institution has been around for a long time does not make its claims more valid or truthful. It can still be questioned. Heresy is still heresy, even if it was introduced early in church history or believed by a majority of professing Christians.
3.) Even if we unanimously agreed to accept Papal authority, that would only eliminate doctrinal conflict in a question begging, tautological sense. That would still not reveal to us whether we should be in communion with the Bishop of Rome (i.e. whether we are right or wrong in our decision making). A case for Roman Catholicism would still need to be made.
4.) Both the Roman Catholic and Eastern Orthodox churches make identical claims of having been established directly by Jesus Christ, but maintain contradictory oral traditions. They dispute each other's claims to authority.
5.) The church fathers sometimes contradicted each other, even themselves. Hence, we have reason to not put them on par with the authority of divine Scripture. These men, no matter how godly or theologically gifted, were not inspired by God. The writings of these people are not to be treated as a smokescreen against what is taught in Scripture. Sometimes church fathers made factual errors. Irenaeus, for example, taught that Jesus Christ lived to be more than fifty years of age, even though gospel tradition indicates otherwise (John 8:57).
8.) We do not have every document written by each church father on every subject. Neither were we present in the early church to take surveys of what everybody believed. This fact alone demonstrates the Roman Catholic claim of unanimous consensus in church history to be vacuous.