He was talking to media persons after attending the memorial reference of Dr Muhammad Aslam Soofi here.
He categorically ruled out any confrontational steps from the government in the aftermath of the JIT report and the Supreme Court verdict.
He said that elections were due in 2018 and Panama case trail was being used as a tool to damage the reputation of the PML-N leadership.
He said that at a time when construction work of China Pakistan Economic Corridor was in full swing, any uncertainty would damage national interests.
He said that the Pakistan Muslim League leadership especially Prime Minister Nawaz Sharif presented himself and his family for accountability for the sake of supremacy of law and Constitution.
He said that the Prime Minister was advised by the legal advisers to question the admissibility and maintainability of the case and challenge the jurisdiction of Pakistani courts, but he did not agree.
The entire nation knows that Nawaz Sharif himself wrote the letter to the Supreme Court for constitution of commission on the Panama papers case and held long consultations with opposition for Terms of References (TORs) of the commission.
But the aim of the opposition was witch hunt of the Sharif family, he added.
Saad Rafique said that all three petitioners of the Panama case were recognized opponents of the PML-N who have been defeated in elections time and again.
When the "Godfather" reference came, the legal advisers asked the Prime Minister to file a reference in Supreme Judicial Council but the government refrained from creating any ill-will.
"We respect all national institutions, but the Parliament and the government are also national institutions and calling Godfather those politicians who get vote of millions of people of Pakistan was not suitable."
He said the PML-N had reservations on inclusion of members of intelligence agencies in the JIT but it kept silent in respect of the superior judiciary.
Referring to the reported WhatsApp call he said they raised the issue but no satisfactory reply came.
Similarly, he said the issue of leakage of Hussain Nawaz's picture arose during appearance before the JIT, and some people tried to blame the government but the JIT admitted its mistake but gave no details.
He said that the JIT also submitted in the court that it had been taping the telephones of the Prime Minister House which was cause of real concern as law of the land does not allow this.
Pakistan has a Constitution and no institution or person is above them all, he said adding all institutions should remain in their constitutional domain.
To a question, he said the JIT should have gone to Qatar to record the statement of Qatari Prime Hamad bin Jassim, as he cannot be treated as Pakistani citizen.
He said a story was published in the leading newspaper about the control of the JIT record and so far no contradiction or clarification has been given from any concerned party.
He said that it was right of the people of Pakistan and the PML-N to know if the story was correct or not.
He said due to the actions and steps of the JIT,the PML-N confidence in it was shaken.
About Imran Khan, Saad Rafiq said he would remain a loser as he has adopted a path which may plunge the country in political upheaval.
He said, "verdicts of the superior court should be implemented irrespective of the fact whether we like them or not."
PML-N would not take any step which may be termed against the national institutions but the requirements of the neutrality should be fulfilled, he added.