We turn to Surah Aa`raaf, it says Musa (AS) chose seventy people out of the group. Musa (AS) then asks God for forgiveness, and begs to God not to punish them based on the fact of what the weak-minded did. So I think this also gives support to the fact that not everybody partook in the act of calf-worship. Not only that after relating the story and following it up with an exhortation to believe in Allah’s final Prophet (S), Allah again speaks about a group from the Israelites who were righteous and people who guided others to the truth.
Also, in Surah Baqarah Allah first speaks about his pardoning the group, and when Hadhrat Musa (AS) returned God forgives them, by prescribing the punishment to ‘kill themselves.’ This punishment, just as in the case of stealing the Shari`ah must be enforced, but this does not absolve the thief of repenting before Allah. I think Allah by pardoning them means he did not destroy them right on the spot, every one of them, because many acknowledged the tragedy of their errors by repenting of what they did. When Hadhrat Musa (AS) returned Allah instituted the punishment of killing themselves, with an exhortation to repent.
But the people who did not partake in the sin were also worthy of blame it seems, as in Surah Aa`raaf, because they failed to check the evils among themselves, allowing the weak-minded to be misled into calf-worship. That is why Hadhrat Musa (AS) and the seventy stood in repentance.
Maybe this could be added to the response, especially the story of Hadhrat Musa (AS) and the elders which occurs in Surah al-Aa`raaf. Maybe it could help in strengthening your position.
But Allah Knows Best.
Thank you for your invaluable comments.