The supreme court has outlawed the use of secret evidence in court by the intelligence services to conceal allegations that detainees were tortured.

The decision will be seen as a significant victory for open justice, but the panel of nine judges pointed out that parliament could change the law to permit such "closed material procedures" in future.

The appeal was brought by lawyers for MI5 seeking to overturn an earlier appeal court ruling that prevented the service from suppressing accusations British suspects had been ill-treated at Guantánamo Bay and other foreign holding centres.

The case arose originally out of claims by Bisher al-Rawi, Binyam Mohamed, Jamil el-Banna, Richard Belmar, Omar Deghayes and Martin Mubanga that MI5 and MI6 aided and abetted their unlawful imprisonment and extraordinary rendition.

The five, who deny involvement in terrorism, launched actions for compensation for abuse and wrongful imprisonment.

The Guardian, on behalf of a number of media organisations, had intervened in the case to argue in favour of open justice.

In the judgment, which runs to nearly 120 pages, all of the judges rejected the security service's main submission that a court has a common law power to order a closed material procedure as an alternative to the more conventional public interest immunity (PII) certificate. Such a power, they argued, would contravene fundamental and long-established principles of open and natural justice.

The court was divided on the security service's secondary submission that a court has a common law power to order a closed material procedure as an add-on to a conventional PII in certain exceptional cases.

Giving his judgment, Lord Dyson said: "There are certain features of a common law trial which are fundamental to our system of justice, both criminal and civil.

"First, subject to certain established and limited exceptions, trials should conducted and judgments given in public. The importance of the open justice principle emphasised many times.

"The open justice principle is not a mere procedural rule. It is a fundamental law principle.

"Secondly, trials are conducted on the principle of natural justice." To allow a "closed procedure" in such an ill-defined way could, he warned, "be the thin end of the wedge".

"This would be a big step for the law to take in view of the fundamental principles at stake. In my view this is a matter for parliament and not the courts."

Binyam Mohamed, Jamil el-Banna, Omar Deghayes and Martin Mubanga. Photograph: Press Association

In similarly forthright terms, Lord Hope dismissed the intelligence agencies' request for legal concealment. "There comes a point," he said, "where the line must be drawn between procedural choices which are regulatory only and procedural choices that affect the very substance of the notion of a fair trial.

"Choices that cut across absolutely fundamental principles – such as the right to be confronted by one's accusers and the right to know the reasons for the outcome – are entirely different. The court has for centuries held the line as the guardian of these fundamental principles."

The Guardian's submissions on open justice were acknowledged by the court. The government has, however, promised to produce a green paper on the use of intelligence material in closed court hearings.

The detainees' claims for compensation have, in the meantime, been settled. The government agreed last November to pay out millions of pounds to former Guantánamo Bay inmates.

The payments followed years of denial by the government of Britain's role in the secret transfer of terror suspects to prisons where they risked being tortured – the CIA practice of "extraordinary rendition".

Responding to the Rawi judgment, Corinna Ferguson, legal officer for Liberty, which also intervened in the case, said: "The government should be humbled by this strong defence of the principles of fair and open justice.

"The law already provides ample protection from disclosure where there are genuine national security concerns, and the court has made it clear that there are no compelling reasons for change.

"We hope ministers will now abandon proposals to introduce yet more secrecy into British courts."

Eric Metcalfe, of the civil rights organisation Justice, which also made legal representations in the cases, said: "The ruling has confirmed that secret evidence has no place in the common law.

"It is a clear setback for the government's plans to extend the use of secret evidence and secret hearings in our courts. Although it is open to parliament to legislate further, today's ruling sets a high hurdle for any MP seeking to cut across centuries of common law tradition."

Clive Stafford Smith, director of the anti-capital punishment charity Repreive, said: "The first casualty in the 'war on terror' certainly was the rule of law, but it is wonderful to see the law fighting back now.

"One must hope that the Conservatives and Liberal Democrats, having been so critical of this kind of infringement of our liberties while in opposition, will maintain that position now they are in power."

In a related judgment issued at the same time, Tariq v the Home Office, the supreme court has ruled by an 8-1 majority that a closed material procedure was permissible in the context of an employment tribunal, where the outcome would be determined by "an independent and impartial tribunal and the disadvantages that the procedure gives rise to will as far as possible be minimised".