On the streets of Tehran in January 1979, a portrait of Ayatollah Ruhollah Khomeini is held aloft. This was the month the shah fled

It was the spring of 1979 and David Owen was Britain’s foreign secretary. He sat one morning in his office off Whitehall and contemplated an event that had just shaken the world.

David Owen’s view

Avoiding the issue of the shah’s odium aided his downfall

The shah of Iran, long the most powerful leader in the Middle East and a firm ally of the US and the UK, had fled his country to escape a storm of popular revolt. Grand Ayatollah Ruhollah Khomeini, the 77-year-old Shia cleric, was assuming the reins of power in Tehran at the head of the biggest mass revolution the world had witnessed since that of Russia 60 years earlier.

Dr Owen, like other senior western policymakers, could not at that moment have imagined the avalanche of events that would follow – the hostage crisis that helped deny Jimmy Carter a second term as US president, the decade-long Iran-Iraq war, the dramatic rise of Islamic fundamentalism that today dominates global security.

Even so, a set of big questions perplexed him. How was it that Britain, like other western states, had failed to predict the fall of a man whom Time magazine had only a few years back called the “Emperor of Oil”? Why was it that British and US policy was for so long closely wedded to the Iranian monarch, ignoring the scale of the political opposition he faced at home? Could a different policy, one that was more frank with the shah about the political threats to his leadership, have saved the regime and changed the course of history?

His concerns prompted him to launch a rare internal inquiry at the Foreign and Commonwealth Office on why Britain and other states had read Iran so wrongly. “I was applying, as a politician, the same discipline that I had lived under as a doctor of medicine where through case conferences . . . the medical profession tries to learn from its mistakes,” says Dr – now Lord – Owen today.

The inquest headed by a senior diplomat, Nicholas Browne, took a year to reach its conclusions. Until now, the 90-page document – marked “secret and confidential” – has been one of the more closely guarded papers in the FCO archive. But it has been seen by the Financial Times and is about to be published under Britain’s 30-year rule, whereby classified documents are released three decades after they were drawn up.

The appearance of a 30-year-old document may not seem much cause for excitement a time when the US is reeling from the publication, via WikiLeaks, of diplomatic cables composed as recently as 10 months ago. Yet the book-length document not only amounts to one of the few internal inquiries that Britain’s foreign office has conducted into a policy failure. It is also striking how devastating its criticism is of UK diplomacy.

On page after page, Sir Nicholas criticises the “failure” of the British embassy – and in particular of Sir Anthony Parsons, then ambassador to Tehran – to predict the demise of the Peacock Throne. “The conclusion that the embassy drew from their analysis [of the shah’s position] consistently proved to be too optimistic,” he says. He makes clear there was failure on every level. In his view, the embassy “overstated the personal popularity of the Shah . . . knew too little about the activities of Khomeini’s followers . . . saw no need to report on the financial activities of leading Iranians . . . failed to foresee that the pace of events would become so fast”. Criticisms such as these are sprinkled through his closely typed pages.

Yet the account is more than just a case study in the failure of diplomatic reporting. Inside the FCO, the document has been studied ever since by a group of senior diplomats for the lessons it offers. “The big questions Browne asks are ones that so many diplomats in the Middle East ask themselves today,” says a senior FCO official. “How can you tell when the country you are reporting on is entering a revolutionary situation? What are the telltale signs to look out for? As diplomats, how do we make sure we are best placed to make calm, objective judgments, even if they are not what our political masters at home want to hear?”

Any analysis of British policy on Iran must start from one core fact: for more than half  a century, many Iranians have suspected Britain of covertly interfering in their country’s affairs. Such suspicion goes back to 1953 when Mohammed Mossadegh, Iran’s elected prime minister, was overthrown in a coup in which US and UK intelligence services were suspected of being protagonists.

As Sir Nicholas writes in his report, “the British gained a reputation for interference in Iran . . . which was remarkably persistent”. Still today, as one FCO official puts it: “Iranians tend to think that [even] if you lift the beard of a mullah you will find a Union Jack underneath.”

Yet as he waded through thousands of diplomatic cables from the period, the picture painted by Sir Nicholas is almost the exact opposite of this myth. In his judgment, UK policy in Iran, far from being confident and scheming, was clumsy and uncertain. As one FCO diplomat who has read the report says: “More often than not, the sense you get is that it was the shah who was running rings round the British, not the other way round.”

Our man in Tehran

Britain and the US may have failed to see the fall of the shah coming. But Sir Nicholas Browne’s report singles out others whose reporting was more accurate – among them the Israeli government. “Their hunches were right when the west’s were wrong,” he notes. Some journalists also come in for praise, including Robert Graham, the FT’s Tehran correspondent from 1975 to 1977. The Foreign Office document says his reporting painted “a more unsavoury and perhaps more realistic picture of Pahlavi’s rule than most of the [British] embassy’s work”.

The embassy’s failures can be seen on numerous fronts. Parsons, who died in 1996, was one of Britain’s most outstanding diplomats. But Sir Nicholas says that as ambassador he failed to glean crucial information needed to do his job. He did not know that the shah was terminally ill with cancer, something that would undermine the monarch’s decisiveness as political opposition to his regime mounted. Parsons (who himself was later frank about the shortcomings) had far too little contact with opposition groups – most notably Khomeini’s followers – for fear that this would upset the shah. As a result, he “underestimated the attractions of [Khomeini’s] simple and consistent message that the shah must be overthrown”.

But the failures were those of diplomacy as well as political reporting. One of Sir Nicholas’s central observations is that British policy in the 1970s – a time of huge economic and political upheaval back home – was heavily driven by the need to sell arms to Iran. As a result, UK and US diplomats held no frank discussion with the Iranian leader about how he should face down internal political challenges until it was too late.

Sir Nicholas reveals in considerable new detail how London set out to appease the shah in the 1970s. On one occasion, diplomats showed the shah the draft of a ministerial answer in the House of Commons on torture in Iran “in case he should object to it”. On another, the FCO “placed” a flattering letter in The Times, describing the shah as “a deeply revered leader”. Most significantly, Britain’s defence ministry indulged in what Sir Nicholas describes as a “helter skelter rush to sell as much as possible to Iran” – when it might have been better to try to persuade him to invest in civil development schemes and quell political unrest.

As a result, it was not until September 1978 – four months before his fall – that Parsons spoke openly to the shah about his political problems. “By then,” writes Sir Nicholas, “most of the damage had been done.”

W ithin the FCO, dozens of diplomats have studied the Browne document as part of their training for the Middle East. One big lesson they have drawn, inevitably, is that when they conduct political reporting, diplomats need to reach out beyond the elites and meet the wider society and opposition movements.

As one former ambassador to Iran puts it: “When members of my staff used to come into my office, I’d always look to see whether their shoes were dirty. If not, I knew they hadn’t been getting out of the embassy and meeting people in town.”

Another lesson inevitably is the danger of allowing arms exports to drive policy so much that political judgments are distorted. For Lord Owen, this is an issue of particular significance. “British foreign policy today suggests we need to be more and more geared up for British exports. There could be no better example of an embassy geared to selling British goods than Tehran in the 1970s. But I say, be careful because you need to get a balanced picture from your embassies.”

Others, meanwhile, would argue that while the Browne report may have been widely read, its lessons have not truly been heeded. Some doubt whether Britain and other western states today do enough to get to know opposition groups in the Middle East. The UK, for example, does not interact with the leadership of the Muslim Brotherhood, the main – but banned – Islamist opposition in Egypt, except through deputies in parliament affiliated with the group.

Ultimately, the document prompts one big question: how should policymakers today judge the future of Iran. The shadow of 1979 still hangs over the regime and especially over Ayatollah Ali Khamenei, Khomeini’s successor. He has long been mindful of how the shah sowed the seeds of his own destruction, repeatedly wavering in his final years in office, giving in to the opposition at times and pulling back at others. As a result, when faced with massive street protests last year, the supreme leader gave not an inch to opponents, refusing any hint of compromise over the election and putting down the unrest.

“Khamenei has learnt the lessons of 1979,” says Ali Ansari, an expert on Iran at St Andrews University. “He’s refusing to budge and he won’t compromise at all because he’s terrified of what happened to the shah.”

Prof Ansari – who has read the Browne document – has a word of warning, however. Like many who have studied the events of those years, he insists that revolutions are inherently unpredictable – and that people should not go around berating those who failed to see a revolution coming. But he argues that the great lesson of the Browne report is that diplomats today should beware of conventional wisdom and in particular the belief that a leader with a strong security apparatus can weather any popular storm.

“Thirty years ago, British diplomats were all guided by a kind of conventional Middle Eastern wisdom that a state with a strong military and a strong security apparatus could survive indefinitely,” he says. “The lesson of Iran in 1979 – and of Nicholas Browne’s report – is that we should not always assume this is how events will turn out.”

Curbs on contact hinder today’s envoys to Tehran

In pre-revolution days, the UK and other western governments were blinded in their analysis by their support for the shah, writes Roula Khalaf. Since 1979, however, their reading of Iran has become more difficult for reasons that are beyond their control.

Washington has had no diplomatic representation in Tehran since the storming of the US embassy there and the ensuing hostage crisis of 1979-80. European governments, even when they have had relatively cordial relations with Iran, have been denied access to the top echelons of power.

Efforts to court opposition forces, meanwhile, have been restricted by a suspicious regime that often brands contact with foreigners as part of a western conspiracy. Britain in particular continues to be deemed the great villain that is constantly conspiring with the US against the Islamic Republic.

While modern communications – in particular blogs – have generated valuable sources of information on life in Iran, the limited direct contact with leading Iranian figures has undermined the ability of diplomats to predict internal changes and to analyse the regime’s strategy in talks about its nuclear programme.

In 2005, the UK and its partners were hoping – and for a long time expecting – that Akbar-Hashemi Rafsanjani, the former president who is considered a pragmatic conservative, would win the presidential election, although he was known to be unpopular. Instead, important parts of the regime lined up behind the fundamentalist Mahmoud Ahmadi-Nejad, propelling him to the second round of voting and establishing him as the favourite of Ayatollah Ali Khamenei, the supreme leader.

While Mr Rafsanjani was seen by many as part of a corrupt elite, Mr Ahmadi-Nejad successfully tapped into social and economic discontent. He also took advantage of disillusionment with the reformists who had controlled the presidency for eight years, helping expand civil liberties but giving insufficient attention to economic development.

It was perhaps the experience of 2005 that led many western diplomats last year to assume – correctly as it turned out – that Ayatollah Khamenei would ensure Mr Ahmadi-Nejad was returned for a second term. But again, diplomats were caught off guard by a shift in public mood and the remarkable revival of the reformist movement, this time led by Mir-Hossein Moussavi, the candidate who was probably the real winner of the election in the first round.

It is the lessons learnt from the pre-revolution diplomatic debacle that are likely to have encouraged western embassies to hire Iranian analysts. During last year’s unrest, however, the authorities arrested several Iranians working for the UK embassy, including Hossein Rassam, its chief political analyst. He was at first charged with espionage but was later given a suspended sentence for acting against the regime.