The White House doesn’t need the support of vocal critics like Sens. John McCain, Lindsey Graham and Kelly Ayotte to confirm United Nations ambassador Susan E. Rice, should the president nominate her as secretary of State. But they almost certainly would need the backing of moderates in both parties, and statements by a number of those senators Wednesday made clear that support is in doubt.

Sens. Susan Collins, R-Maine, and Bob Corker, R-Tenn. — both swing votes on foreign policy issues — issued downbeat assessments of their meetings with Rice on Capitol Hill on Wednesday.

Neither lawmaker took a firm position — as McCain, R-Ariz., Graham, R-S.C., and Ayotte, R-N.H. did Tuesday — on opposing a Rice nomination. But both Collins and Corker expressed grave doubts about the Obama administration’s handling of the terrorist attack on the U.S. consulate in Benghazi, Libya, in September and the role Rice played in the response.

Their outlook, shared by other more moderate GOP senators, as well as a refusal by some Senate Democrats to offer their support, points to a steep uphill climb for the White House should the president decide he wants to take on the Hill and nominate Rice, who is believed to be his preferred pick to succeed Hillary Rodham Clinton.

Graham and Ayotte confirmed Tuesday after their own meeting with Rice that they would try to block her nomination until they get more answers about the Benghazi attack, which killed the U.S. ambassador, J. Christopher Stevens, and three other Americans. Democrats, however, could try to defeat any filibuster of Rice with 60 votes in the Senate, and would only need five Republican votes, in addition to the two independents expected to caucus with them, in next year’s Senate to reach that threshold.

Getting those five votes will be no easy task, however, nor is there any guarantee Democrats will maintain party unity on a Rice nomination. While some Democratic senators, including Majority Leader Harry Reid of Nevada and Armed Services Chairman Carl Levin of Michigan, have made clear that they think Rice is qualified to be America’s top diplomat, others are reserving judgement.

“I reserve my perspective based on an interview that I’m sure, if she gets nominated, will occur during the confirmation process,” Jon Tester, fresh off a victory in a hotly contested Montana Senate race, said Wednesday.

Another swing state Democrat, Joe Manchin III of West Virginia, quickly shifted the subject to another potential candidate. “I don’t know much about it,” he said of the controversy over Rice, “but I can tell you I’m sure [Senate Foreign Relations Chairman] John Kerry is someone I think would be very, very good.”

Sen. Robert Menendez, D-N.J., also wasn’t ready to give the U.N. ambassador his endorsement. “I haven’t really looked at Ambassador Rice as a nominee because she isn’t a nominee. When and if she becomes a nominee, then I’ll look at it,”he said.

On the Republican side, the White House is looking at a shrinking pool of swing votes with the departure of New England moderates such as Olympia J. Snowe of Maine and Scott P. Brown of Massachusetts. And Collins, who also falls in that category, gave the administration little reason for optimism Wednesday.

“I continue to be troubled by the fact that the U.N. ambassador decided to play what was essentially a political role at the height of a contentious presidential election campaign by agreeing to go on the Sunday shows to present the administration’s position” on Benghazi, she told reporters following her meeting, echoing the key issue Republicans have raised in their critique of Rice.

Specifically, critics have zeroed in on Rice’s statements on several Sunday morning news programs five days after the attack in which she claimed that the assault appeared to have been a spontaneous response to an anti-Muslim video. Intelligence officials now say that the assault was a coordinated effort by a fundamentalist Islamic militia, but they also have maintained that Rice was only repeating the unclassified talking points they provided officials at the time. President Barack Obama has fiercely defended Rice, a longtime foreign policy adviser, saying at a cabinet meeting Wednesday, “Susan Rice is extraordinary.”

“I couldn’t be prouder of the job that she’s done,” the president added.

Corker, who is poised to be the ranking Republican on the Senate Foreign Relations Committee next Congress, also complained Wednesday about the amount of political spin in the Obama administration’s communications with Congress on the Benghazi attack.

“I would just ask the president to step back for a moment and realize that all of here hold the secretary of State to a very different standard than most cabinet members,” he told reporters after his meeting with Rice. “We want someone of independence, someone — we understand that they’re going to support the administration and their efforts — but someone who’s transparent and direct.”

And while Corker stressed he was not speaking about Rice in particular and has yet to pass judgment on her potential nomination, he has left little doubt in recent days that he does not think she fits any of those categories.

Other Republicans the White House could court to get to 60 votes voiced their neutrality Wednesday, but the factors they said would be key to their decisions on a Rice nomination could prove problematic for her.

Veteran Sen. Lamar Alexander, R-Tenn., said that while he would evaluate Rice’s nomination carefully, should the president tap her, he would be influenced by those lawmakers who have been sounding alarms about Benghazi. “I have great respect for Sens. Collins, Ayotte and Graham and McCain, and if they have a strong opinion, which they appear to have here, that’s going to make a difference in my thinking,” Alexander said.

And Sen. Dan Coats, R-Ind., a one-time ambassador to Germany, said the administration first needs to clear up all the outstanding questions he and other lawmakers have “relative to everything that happened in Benghazi, how it was portrayed by the administration and so forth,” before he could consider a potential Rice nomination.

Added Coats: “The one has to precede the other.”