Well they may have even denied the infallibility of the Prophets. And it is also possible that they didn't understand the Verses in the Qur'an which some Shi'a say prove infallibility.
Sometimes the tafsir of the Imams is needed.
So please share with us the Tafseer of those verses which Shia use for infallibility from the reliable sayings of Imam.
I like this hadith on wilaya:
`Ali b. Ibrahim from his father from ibn Abi `Umayr from `Umar b. Udhayna from Zurara, al-Fudayl b. Yasar, Bukayr b. A`yan, Muhammad b. Muslim, Burayd b. Mu`awiya and Abu’l Jarud together from Abu Ja`far عليه السلام.
He said: The wilaya of `Ali was an order from Allah عز وجل to His messenger, and He revealed upon him, “Verily your guardian is Allah, His messenger, and those who believed – who stand in prayer and give the zakat…” (5:55). And He made the wilaya of the Possessors of the Command (ulu ‘l-amr) obligatory, and they did not understand what it was; so Allah ordered Muhammad صلى الله عليه وآله to interpret [and clarify] the wilaya to them, just as he interpreted salat, zakat, sawm, and Hajj. When Allah gave that [order] to him, the chest of the Messenger of Allah صلى الله عليه وآله tightened, and he feared that they would apostatize from their religion and bely him – so his chest tightened. He consulted his Lord عز وجل, so Allah عز وجل inspired to him, “O Messenger, preach what is revealed to you from your Lord. If you do not preach, it will be as though you have not conveyed My Message, and Allah will protect you from the people”. (5:67) So he executed the command of Allah تعالى ذكره and declared the wilaya of `Ali عليه السلام on the day of Ghadeer Khumm. He called for a congregational prayer and commanded the people to bear testimony and inform the absent.
`Umar b. Udhayna said: All except Abu’l Jarud said:
And Abu Ja`far عليه السلام said: And an obligation would be revealed after the other, and the wilaya was the final obligation, so Allah عز وجل revealed, “Today I have perfected for you your religion and completed my favour…” (5:3). Abu Ja`far عليه السلام said: Allah عز وجل says: I will not reveal to you any obligation after this – I have completed for you the obligations. (al-Kafi, Volume 1, hadith 753)
(hasan) (حسن)
Hadith taken from ShiaChat, translated by a brother on there.