Salam
The Quran is interconnected and protects the meaning of the verses and their intended places in Quran by other verses.
"And we did not send before you except men who we revealed to, therefore ask the family of the reminder if you do not know"
This verse unites a lot of verses in Quran and brings together a lot of themes into one.
The Messenger is a reminder and the clear proof
The Quran mentions that the Messenger is a reminder that Allah (swt) has brought down. In Suratal Bayana, it further states the Messenger is the clear proof.
In this regard, regarding being the clear proof, the Quran reminds when people are making demands for a sign being brought down upon him that he is but warner and for every people there is a guide.
The question is who was the clear proof before Mohammad? And who is the clear proof after?
This verse is saying the reminder's family is to be asked. The Quran also states the book of Moses was a leader before, and we know this was through leaders who guide by God's command. And hence the book of Moses and the leaders who guide by God's command were one.
We see again this message of the two weights left by the Prophet found in this verse. The reminder's family are themselves each an instance of the reminder. Their leadership is the leadership of the reminder, and it's one and the same, and the means by which Quran becomes a leader among humanity is linked with asking them and seeking knowledge from them.
For the sake of argument, say Mohammad didn't know!
There is many verses about if Mohammad was in doubt or didn't know the truth fully, what he should do in that case.
Among them is the verse saying to ask the Messengers sent before him if they ever commanded to worship a god with God, and among them is the verse stating if he was in doubt to ask those given the book before him.
This verse is similar in some regards is saying say for the sake of argument Mohammad lacked knowledge and was not a Messenger, what would his people be required to do then? What would he be required to do?
The Quran guides in this instance, there surely would be a reminder from God and it surely would have a family. If it was not Mohammad and Quran that was the reminder, and not his family, there surely would be a reminder and it would have a family of guides to be asked and sought knowledge from.
The theme of chosen families
Here it truly reminds us the whole wisdom of why families and the book, the rope from God and rope from humans, has been emphasized throughout Quran.
That all those sent before were sent with reminders and were groups known as "family".
This is not the first reminder and not the first family that has been interlinked with the reminders of God.
But as for a reminder and a family, the Quran has shown eternal wisdom in this way, and has emphasized on these Captains that sail believers to safety by the name of God.
The reminder and the family together are the ship of salvation.
There is no doubt such a ship, the only question remains, who exactly is that instance.
Why it doesn't make sense it's people of the book
One is because it didn't say people of the book and reminder in the two Surahs it mentions particularly refers to the newest reminder (ie. the remainder of this current time).
Why it doesn't make sense to refer to normal fallible scholars
Aside from the warning from much of the Quran to be warry of religious leaders who have no authority from God and not to rely on them to think for you, it is obvious such people don't have all the answers to our questions, and is unrelated to the fact that God sent only men who he revealed to.
Aside from that, if Mohammad didn't know, would he be required to seek knowledge from which scholars exactly? Rather, he must seek to know the current reminder of God and it's designated family as the whole Quran shows the wisdom of God revealing reminders and designating a chosen family for it.
The religious scholars always in each era when they took the mantle of authorities to be followed and streams to be relied on, lead people away from the reminder and it's family.
Context shows it to mean the family of Mohammad
Surat Anbiya begins to talk about a new reminder and then after this verse emphasizes on chosen ones guiding in the past. Particularly it reminds of the Abraham, Isaac, and Jacob of being leaders who guide by God's command and that God revealed them actions of goodness and prayer(connection) and that it was God that they worshiped. As for the last statement, it is to emphasize as for the leaders who truly worship God and value him above all else, then they cannot be known by our whims and simply calling who we want righteous.
Rather God shows through the axis of his reminder and chosen ones like Abraham, Isaac, and Jacob, who were the "Ahlulbayt" of the time, that God manifests the truth and it's followers through them.
Scholars of all religions will always mantle of sincerity, piety, and purity, but we know at the end, God wants us to rely on the chosen ones who come in groups and are the chosen family of the reminder of the time.
In Surat al Nahl it emphasise that reminder has been revealed to Mohammad so that he may manifest and explain what has been revealed to them.
The context shows his family is meant to take on this essential role of manifesting what has been revealed to God to the people. They would become the speaking Quran's that manifest the Quran and explain it properly, as well, as the Sunnah.