LONDON (AP) — They were two very talented brothers with so much potential. The older one was poised to become the leader of Britain’s Labour Party, with a chance to become prime minister, while the younger was expected to rise with his brother to the highest ranks of the country’s political arena.

But things didn’t go that way for David and Ed Miliband. Ed didn’t want to take a back seat to his more polished and articulate older brother and shocked Britain’s establishment in 2010 by challenging David for the Labour leadership role and triumphing.

Five years later, that victory has turned bitter.

The Downing Street dreams of 45-year-old Ed have been dashed by his party’s dismal failure in Thursday’s election, which saw Prime Minister David Cameron’s Conservatives score a convincing victory over a weakened Labour Party.

The Miliband family has suffered as well: A gulf as wide as the Atlantic Ocean has opened between the once-close brothers, with David, 49, abandoning politics and moving to self-imposed exile in New York. That has left the brothers’ 80-year-old mother, a Holocaust survivor, trying to bridge the gap.

John Rentoul, author of a biography about former Labour Prime Minister Tony Blair and a columnist at the Independent on Sunday, believes the leadership contest has split the brothers, perhaps forever.

“They have to maintain a good face in public,” he said. “But I don’t think their relationship is easy. I do not see how David could ever forgive his brother for what he did.”

Yet the last few days have seen rumblings that David might return to Britain to jumpstart his political career and try once more to become Labour party leader after Ed resigned the post Friday. The prospect was fuelled by the Twitter musings of one of Labour’s best known and most wealthy donors, Harry Potter author J.K. Rowling, who admitted she was obsessing about David.

The obsession goes like this: Ed, for all his earnestness, proved a dismal campaigner, and many believe David would have been far more formidable. Some Labour supporters suffer from “what might have been” syndrome, a regretful sense that the party, heavily influenced by unions during the 2010 leadership contest, picked the wrong brother.

Rentoul says he believes David would probably have had the political skills to win Thursday’s election, in part by following Blair’s “New Labour” strategy of positioning the party at the country’s political center.

“They should have chosen David Miliband last time,” he said. “Now they have to learn their lesson and choose someone who would follow the ‘New Labour’ path even if they don’t call it that. Ed was imposed on the party by the union bosses who mobilized the activist base to swing it for Ed and move it to the hard left. That was a disastrous intervention.”

David has left the door slightly open to a possible return someday, but he has given no indication he will jump into the upcoming leadership race, which is already drawing a number of youthful Labour Party hopefuls. He did say via Twitter that his “heart goes out” to the many Labour colleagues who lost their seats — and to Ed.

Despite Thursday’s setback, the larger Miliband family saga is one of triumph over extreme adversity.

Their parents were Jewish refugees who barely escaped the Nazi push through Europe during World War II. Their late father, Ralph Miliband, was able to board one of the last boats to leave Belgium before the Nazis took control, finding safety on British shores. Their mother, Marion, narrowly survived the Nazi invasion of Poland and found refuge in Britain after the war. Many of their relatives died in the concentration camps.

The Milibands flourished in their new home. Ralph Miliband, after enlisting in the British military to fight the Nazis, became a leading Marxist writer and professor.  Marion was also active in leftist politics, and their two sons did extremely well at Oxford University before moving adroitly up the political ranks.

David became foreign secretary in Gordon Brown’s 2007-2010 Labour government, hobnobbing with world leaders like Hilary Clinton, who as U.S. Secretary of State described David as “vibrant, vital, attractive (and) smart.” Ed impressed in lesser Cabinet roles as head of the Energy and Climate Change department.

While David stuck closely to Blair’s centrist model, based on a middle-of-the-road policy that kept business backers on board, Ed sought to return Labour to its working class, union roots, backed by Brown. 

Ed’s decision to challenge David for the party helm violated the traditional etiquette in political families, as exemplified in the Kennedy clan, which saw Bobby Kennedy work tirelessly to help his older brother John Kennedy win the presidency, only seeking the top job after his brother was assassinated. Teddy Kennedy then supported older brother Bobby until he too was shot down.

Some who take a Freudian view of things believe Ed was, consciously or not, trying to please his dead father by sticking closer to his ideals. Psychotherapist Oliver James, author of “Affluenza,” a book about the unhappiness plaguing many wealthy people, said Ed is more in touch with his late father’s ideology than David.

“In his heart and mind he was looking for ways to get approval from his father’s ghost by remaining truer to his father’s belief,” James said. “And he was telling his older brother: ‘Actually, mate, I’ve got just as much right to run for this as you do.’ He was saying: ‘I’m just as good as you, if not better.”’

The ultimate impact of their sibling rivalry on Britain is not clear. David, who heads the International Rescue Committee in New York, has attractive options, and Ed may, like many politicians before him, rebound from his defeat.

The Labour Party’s contingent in Parliament is now smaller than it was when Ed took over — but some believe that is not his fault.

Ann Clywd, a longtime Labour Party legislator who won re-election in Wales, said Labour’s fall resulted from the tremendous surge of support for the independence-minded Scottish National Party that cost Labour dozens of seats in a traditional stronghold.

“I don’t think it was a matter of Ed Miliband,” she said. “I actually supported the other one, but Ed was the better man for the job. He was very good with people, he listened to people. People had warmed to him actually. I think it was the Scottish factor.”