By Ray Sanchez, Elise Labott and Patrick Oppmann, CNN

(CNN) — Alan Gross, a U.S. government subcontractor imprisoned in Cuba for smuggling satellite equipment onto the island, is being held at Havana’s Carlos J. Finlay Military Hospital.

With peeling canary-yellow walls and hordes of people coming and going, the aging building doesn’t look like a place where Cuba would hold its most valuable prisoner.

But police officers and soldiers surround the hospital. Inside, Cuban special forces guard the 65-year-old U.S. citizen, emotionally and physically frail and approaching his fifth year in confinement.

North of the Florida Straits, Gross’ imprisonment is seen as the major impediment to better relations with Havana.

Now, however, midway through the second term of President Barack Obama, several signs of possible change have emerged. Senior administration officials and Cuba observers say reforms on the island and changing attitudes in the United States have created an opening for improved relations.

The signs include the admission this week by senior administration officials that talks about a swap between Gross and three imprisoned Cuban agents — part of group originally known as the Cuban Five — have taken place. In addition, recent editorials in The New York Times have recommended an end to the longstanding U.S. embargo against Cuba and even a prisoner swap for Gross.

Who is Alan Gross?

U.S. officials said Gross, who is Jewish, was trying to help Cuba’s small Jewish community bypass stringent restrictions on Internet access.

Cuban authorities, however, countered that he was part of a plot to create a “Cuban Spring” and destabilize the island’s single-party Communist system in a clandestine effort to expand Internet access. Gross had traveled to Cuba multiple times as a tourist.

Gross had worked for Development Alternatives Inc., a Maryland-based subcontractor that received a multimillion-dollar U.S. contract for so-called democracy building on the island.

Fulton Armstrong was a senior adviser to the Senate Foreign Relations Committee under then-Sen. John Kerry when Gross was arrested. The subcontractor’s mission, under what Armstrong characterized as USAID “regime-change” programs, was “dangerous and counterproductive,” he said.

Alan Gross ‘withdrawn,’ saying goodbye

A 2012 lawsuit filed by Gross’ wife, Judy, accused USAID and Development Alternatives Inc. of negligence. It said the agencies had a contract “to establish operations supporting the creation of a USAID Mission” in Cuba.

The operation involved the smuggling of parabolic satellite dishes hidden in Styrofoam boogie boards, Armstrong said. Cash was transported to Cuba to finance demonstrations against the Castro regime.

“They were sending this poor guy into one of the most sophisticated counterintelligence operating environments in the world,” said Armstrong, who spent 25 years as a CIA officer. “It was not credible his story about the Jews. It didn’t make sense.”

In March 2011, Gross was tried behind closed doors for two days and convicted of attempting to set up an Internet network for Cuban dissidents “to promote destabilizing activities and subvert constitutional order.”

Wife of Alan Gross says he feels ‘hopeless’

Gross’ lawyer, Scott Gilbert, said years of confinement have taken a toll. His client has lost more than 100 pounds. He is losing his teeth. Gross’ hips are so weak that he can barely walk.

Gross, who has lost vision in one eye, has threatened to take his life, Gilbert said. Frustrated with the lack of progress in his case, the American has refused to see U.S. diplomats who once visited him at least monthly.

“Emotionally, Alan is done,” Gilbert said. “He said goodbye to his family in July. … He has prepared himself, as he has said, to come back to the United States, dead or alive. Time is very short.”

Who are the Cuban Five?

The name may conjure images of the tropical equivalent of the Jackson 5, but the Cuban Five are agents convicted in 2001 for intelligence gathering in Miami. They were part of what was called the Wasp Network, which collected intelligence on prominent Cuban-American exile leaders and U.S. military bases.

The five — Ruben Campa, also known as Fernando Gonzalez; Rene Gonzalez; Gerardo Hernandez; Luis Medina, also known as Ramon Labanino; and Antonio Guerrero — were arrested in September 1998.

Hernandez, the group leader, also was convicted of conspiracy to commit murder for engineering the downing of two planes flown by the exile group Brothers to the Rescue in 1996. He’s serving two life sentences.

Cuban fighter jets shot down the unarmed Cessnas as they flew toward the island, where they had previously dropped anti-government leaflets. Four men died.

At trial, the defendants said their mission was to gather intelligence in Miami to defend Cuba from anti-Castro groups they feared would attack the island. Seven members of the network cooperated with U.S. authorities and are believed to be in witness protection.

In February, Fernando Gonzalez was released from a U.S. federal prison after serving 15 years for failing to register as a foreign agent and possessing forged documents.

In 2011, Rene Gonzalez was released after serving most of his 15-year sentence.

In Cuba, the two spies were welcomed as heroes. They were considered “political prisoners” unjustly punished in American courts. Their faces appeared on billboards throughout the island. State-controlled media labeled them “terrorism fighters.”

A federal appeals court originally threw out their convictions but later reinstated them.

Defense lawyers accused lower courts of unfairly refusing to move the trial to Fort Lauderdale, Florida, from politically charged Miami, where anti-Castro hostility was more prevalent. They also raised serious questions about the jury selection process.

The trial for the Cuban Five was the only judicial proceeding in U.S. history condemned by the U.N. Human Rights Commission. Amnesty International also raised serious doubts about the fairness and impartiality of their trial.

Why do some people believe the time is right for a swap?

With the U.S. midterm elections over, some Cuba observers believe the time is ripe for a breakthrough in relations. As a second-term president, Obama doesn’t have to worry about re-election.

“The political stars are well aligned because both Obama and (Cuban leader) Raul Castro have repeatedly said that they’d like to see an improvement in relations,” said William LeoGrande, an American University professor and co-author of a new book, “Back Channel to Cuba,” which chronicles decades of negotiations between the two countries.

In April 2015, at the Summit of the Americas in Panama, the two leaders may have an opportunity to meet face to face.

Before then, the White House can lay the groundwork for agreements aimed at “burying the historical hatchet between the U.S. and Cuba,” said Peter Kornbluh, co-author of “Back Channel to Cuba” and senior analyst at the National Security Archive.

“Richard Nixon went all the way to China, and Barack Obama only has to go to Panama,” he said.

In Washington, senior administration officials predict more cooperation, with an important caveat.

“There is stuff we can do, but it has to start with Gross,” one of the officials said.

Administration officials say talks about a possible swap have taken place, but they’re hesitant to speak about whether those discussions are progressing. The White House came under fire after the recent swap of five Taliban detainees for the release of U.S. Army Sgt. Bowe Bergdahl.

Bergdahl was freed last spring after nearly five years in captivity at the hands of militants in Afghanistan. His controversial release came in exchange for five mid- to high-level Taliban detainees from Guantanamo Bay Naval Base in Cuba.

“They’ve been in jail for 16 years,” LeoGrande said of the Cuban agents, “and on humanitarian grounds alone it’s reasonable to release them when we stand to gain the release of an American citizen.

“It’s a better deal than trading five Taliban commanders for one U.S. soldier.”

No one knows how the incoming Republican-controlled Senate will handle Cuba policy. Most Republicans don’t feel strongly about the Cuba issue, and some lawmakers in agricultural states have supported a lifting of the trade and financial embargo in force for more than 50 years.

With Sen. Robert Menendez, D-New Jersey, soon to be replaced as chairman of the Senate Foreign Relations Committee, one of the most powerful opponents to greater engagement with Cuba will have a decreased platform from which to criticize the administration on Cuba issues. But Menendez will remain on the committee, as will Sen. Marco Rubio, R-Florida, another strong Cuba critic.

Why do others say the swap won’t happen?

On Tuesday, White House press secretary Josh Earnest told reporters that while the President has said Cuba policy is worth reconsidering, the administration has “significant concerns … about (the Cuban government’s) human rights record, their failure to observe basic human rights, as it relates to not just the illegitimate detention of Mr. Gross, but as it relates to the basic rights to free speech and political expression of the people of Cuba.”

Some longtime Cuba observers are skeptical of the prisoner-swap idea.

“It’s conceivable that it could happen now,” said Armstrong, the former senior adviser to the Senate Foreign Relations Committee. “Who knows? (Attorney General) Eric Holder is leaving the and Obama is now pretty much a lame duck, and Bob Menendez will no longer be chairman of foreign relations, and Alan Gross should be home by Thanksgiving or Christmas or Hanukkah. Enough is enough. But we’ve been at this point before.”

In “Back Channel to Cuba,” LeoGrande and Kornbluh describe backdoor negotiations in 1963 that led to the release of more than two dozen Americans jailed in Cuba, including members of a CIA team caught planting listening devices in Havana.

The U.S. gave up four Cuban prisoners, including an attaché at the U.N. mission and two indicted for planning acts of sabotage. The fourth was a Cuban convicted of murder for killing a 9-year-old girl who was struck by a stray bullet during a fight with anti-Castro Cubans when Fidel Castro visited New York in 1960.

Castro granted clemency to the American prisoners. And the United States released the Cubans in what the Justice Department described as an act of clemency “in the national interest.”

In 1979, President Jimmy Carter granted clemency and released three Puerto Rican nationalists, including Lolita Lebron, who had been convicted for opening fire in the U.S. House of Representatives and wounding five congressmen. The deal was part of a backdoor “humanitarian exchange” in which Fidel Castro released four CIA agents 11 days later.

Said Kornbluh, “It is time to bring U.S.-Cuba relations into the 21st century.”

CNN’s Ray Sanchez reported and wrote in New York, Elise Labott reported from Washington and Patrick Oppmann reported from Havan