You see this Verse (2:124) is not talking about the issue of Ismah "infallibility". And let me be honest, when Shi'ites quote this Verse (2:124), they aren't trying to prove the infallibility of Imams, they are quoting this Verse as a proof for the fact that Allah is the One who makes the Imam, i.e., their doctrine that the Imam is divinely appointed by Allah just as the Prophet is divinely appointed.
Now if you as a Sunni take this Verse (2:124) and say that the covenant mentioned in it is only concerning Prophets, I find that dishonest, because it is clear that you are saying that only to oppose the Shi'ite doctrine, and so you have a preconceived bias in approaching the text of the Holy Qur'an.
Now I will clarify my doctrine. According to me Allah does indeed give Imamate to the progeny of Abraham with a Nass, i.e., directly naming and appointing an Imam to lead and guide the people. This is the Sunnah of Allah. And those whom Allah appoints as Imam are not always Prophets. I will give some examples:
I've already mentioned the example of Talut عليه السلام
وَقَالَ لَهُمْ نَبِيُّهُمْ إِنَّ اللَّـهَ قَدْ بَعَثَ لَكُمْ طَالُوتَ مَلِكًا
And their Prophet said to them, "Indeed, Allah has sent to you Talut as a king." (2:247)
وَلَقَدْ أَخَذَ اللَّـهُ مِيثَاقَ بَنِي إِسْرَائِيلَ وَبَعَثْنَا مِنْهُمُ اثْنَيْ عَشَرَ نَقِيبًا
And Allah had already taken a covenant from the Children of Israel, and We delegated from among them twelve leaders. (5:12)
So you should know that King Talut (Saul) and the 12 Nuqaba' (Reuben, Simeon, Levi, Judah, Issachar, Zebulun, Dan, Naphtali, Gad, Asher, Benjamin, and Joseph) were not Prophets, with the exception of Joseph.
So it is proven that Allah Most High did indeed appoint non-Prophet to the position of Imamate with a Nass, i.e., through Wahi. They did not acquire Imamate through consultation and election by the people but were divinely appointed by Allah.
Now if you say that these are examples from the previous Ummam, then I will give you an example from this Ummah. That is the example of Allah appointed the Mahdi to be the Imam. The Mahdi's Imamate has a Nass from the Prophetic Hadith:
عَنِ النَّبِيِّ صلى الله عليه وسلم قَالَ " لَوْ لَمْ يَبْقَ مِنَ الدُّنْيَا إِلاَّ يَوْمٌ " . قَالَ زَائِدَةُ فِي حَدِيثِهِ " لَطَوَّلَ اللَّهُ ذَلِكَ الْيَوْمَ " . ثُمَّ اتَّفَقُوا " حَتَّى يَبْعَثَ فِيهِ رَجُلاً مِنِّي " . أَوْ " مِنْ أَهْلِ بَيْتِي يُوَاطِئُ اسْمُهُ اسْمِي وَاسْمُ أَبِيهِ اسْمَ أَبِي " . زَادَ فِي حَدِيثِ فِطْرٍ " يَمْلأُ الأَرْضَ قِسْطًا وَعَدْلاً كَمَا مُلِئَتْ ظُلْمًا وَجَوْرًا
The Prophet (ﷺ) said: If only one day of this world remained. Allah would lengthen that day (according to the version of Za'idah), till He raised up in it a man who belongs to me or to my family whose father's name is the same as my father's, who will fill the earth with equity and justice as it has been filled with oppression and tyranny
Hattaa Yab'atha Feehi Rajulam Min Ahli Baitee "Until He (Allah) raised up in it a man (Mahdi) from my Household" (Sunan Abi Dawud: Isnaduhu Hasan)
So it is known that the Mahdi is an Imam who is directly appointed by Allah with a Nass; we as the Umma don't have the authority to consult among ourselves and elect our own Mahdi.
Now where we differ with the Shi'ites is concerning a specific issue, which is whether the Prophet ﷺ directly named his successor with a Nass, and whether their 12 Imams are divinely appointed Imams. This is the extent of our difference with them. To argue with them over the theory of divinely-appointed Imamate and to argue that there is no such concept as a divinely-appointed Imamate is to shoot oneself in the foot. In such a debate the Shi'ite will always win. Hope you understand where I'm coming from now