Many lawmakers from both parties say they consider the extramarital affair of David H. Petraeus and his retirement from the CIA a distraction from a more meaningful spy-related investigation: what happened in the lead-up to and fallout from the September terrorist attack on the U.S. consulate in Benghazi, Libya.

The House and Senate panels are seeking testimony from Petraeus on Benghazi. Senate Intelligence Chairwoman Dianne Feinstein, D-Calif., confirmed Wednesday that Petraeus has agreed to testify before her panel about Benghazi, although she was not certain of the timing. And House Intelligence Committee officials announced Wednesday evening that Petraeus would testify in closed session Friday morning about the events in Libya.

“I think there’s a lot of interest in talking to Gen. Petraeus and getting his Benghazi report, not this other thing,” said Sen. Roy Blunt, R-Mo., referring to a Petraeus-written report on his recent trip to Libya. Blunt is a member of the Intelligence Committee.

“That’s the most focus — what did you learn in Benghazi and what do we need to know about it?” he said.

Although a number of lawmakers have serious concerns about whether Petraeus’ affair has national security ramifications, some view it more as a media-orchestrated scandal of lesser consequence than other matters.

“Look, the media coverage obviously is on these new emerging facts in the Petraeus and other cases,” Rep. Chris Van Hollen, D-Md., told MSNBC on Tuesday, agreeing that the issue was taking up a lot of “oxygen” that might otherwise be devoted to Congress’ efforts to avoid the fiscal cliff of tax increases and program cutbacks. One of the “other cases” includes an inspector general investigation into Marine Corps Gen. John R. Allen Jr., the current commander of all allied forces in Afghanistan, regarding a relationship with a woman tied to the Petraeus scandal.

Feinstein on Tuesday largely shied away from answering questions about Petraeus’ affair, saying that her focus was on Benghazi. Her panel’s vice chairman, Saxby Chambliss, R-Ga., also said Tuesday that he is more interested in Benghazi, adding that some of it was practical in nature; the FBI has an ongoing investigation into Petraeus’ personal behavior, while only Congress can dig into the attack on the consulate in Benghazi, he said.

Senate Armed Services ranking Republican John McCain of Arizona said Wednesday that the disclosures about Petraeus are “very serious, and I think it deserves a thorough and complete investigation. But it does not rise to the level of an attack on an American consulate that took four American lives.”

McCain and GOP colleagues Lindsey Graham of South Carolina and Kelly Ayotte of New Hampshire introduced a resolution Wednesday calling for the creation of a bipartisan, bicameral special committee to investigate Benghazi to replace the multiple, stove-piped investigations that various committees on both sides of the Capitol are currently conducting.

“We believe the complexity and gravity of this matter warrant the establishment of a temporary select committee that can conduct an integrated review of the many national security issues involved, which cut across multiple executive agencies and legislative committees,” McCain said at a press conference announcing the resolution.

Feinstein, however, quickly rejected the proposal after appearing open to it earlier in the week.

“We have arranged for three hearings on the intelligence aspects of it: the intelligence aspects of the actual attack, the intelligence aspects of the security and the intelligence aspects as to adequacy in the Middle East,” she said. “So we will be handling the intelligence part of it. We are the committee that should do it. We are the committee that is responsible for oversight and has the authority to do it. I see no need for a special committee.”

McCain also said that he and other members of the Senate Armed Services panel “have an interest in what Petraeus has to say” about Benghazi. Graham added that the panel also expects to hear from Defense Secretary Leon E. Panetta as well as retiring Africa Command chief Gen. Carter F. Ham, who was in charge of U.S. military operations in the region at the time of the Benghazi attack.

Armed Services Chairman Sen. Carl Levin, D-Mich., said Wednesday that he did not want to hold any hearings on Benghazi until after the State Department completes its own review, due in mid-December.

“I don’t want to contribute to any political pressure or any suggestion of wrongdoing,” Levin said.

President Barack Obama on Wednesday repeated his pledge to investigate the attack fully and keep Congress informed.

“It is important to find out what happened in Benghazi,” he said at a news conference. “There needs to be accountability.”

When it comes to the Petraeus scandal, Graham said it is important that lawmakers “segregate out the weird from the national security.”

“This story is getting weirder by the day,” he said. “There is the weird and the strange and the human failings in one camp, and there is a legitimate question about national security being breached in the other camp.”

“At the end of the day,” Graham continued, “who knows where this thing is going to go?”

Jennifer Scholtes, Niels Lesniewski and Matt Fuller contributed to this story.