To be just and fair, there is no such thing as absolute consensus, there will ALWAYS be someone with a different opinion and that is human nature BUT when we say consensus, we mean at least the vast majority.
At the beginning there was no consensus simply because nobody was directly appointed, but the 48 hours after the death of the Prophet (saw) showed us that the vast majority of those who remained on Islam, especially the dwellers of Makkah and Madinah who were the people who knew most about Islam and were exposed to it more than some wondering Arabs in the middle of the desert, at least 98% of them pledged loyalty to Abu Bakr and stuck to it.
These numbers were preserved for `Umar after him, then `Umar allowed six people to consult and figure things out for themselves, and various tribes had favored various candidates, but the same number was still preserved although divided between the six. After `Uthman acquired authority through popularity, the other five candidates and all those who support them followed him.
Some people complained in the last years of `Uthman's life, yet everyone was united behind him, until a group of ignorant rebels decided to force him to resign and accused him of illegitimate things, this happened in the last couple of months of his life, this rebel group was not very large and the people of Madinah alone could have beaten them, but `Uthman ordered otherwise.
When `Uthman was killed, that was the first real split in Islam, and then the Fitnah struck, the majority still believed `Ali to be legitimate Caliph, but the Shamis rejected him and accused him of the previous Caliph's assassination, a lot of Sahabah decided to remain on the sidelines until the dust cleared and refused to fight against other Muslims, that split was followed by another when a small group of `Ali's Shia turned on him and separated, still it was only the Caliph fighting against a rebellious governor and another group of Takfeeri rebels.
After this, al-Hasan who had full authority decided to step down and offer it to banu Umayyah who unified the lands and stopped every rebellion until banu al-`Abbas.
Matter of the fact is, whether you reject Abu Bakr's Caliphate or `Ali's or Mu`awiyah's, it really changes nothing today, since they're dead and don't care and they had authority and ruled when they were alive and succeeded in doing great things.
Allah will not ask you about Abu Bakr's authority, nor will he give you history test, but you WILL be asked whether every word you utter and every action you make is for the sake of Islamic brotherhood and unity or towards unnecessary division and Fitnah.