House Education and Labor Chairman George Miller (D-Calif.) said Friday that top House Democrats need to weigh whether Rep. Charlie Rangel (D-N.Y.) can continue to effectively lead the Ways and Means Committee.

The ethics committee disclosed Thursday that it had admonished Rangel for taking corporate-paid trips to the Caribbean.

“It makes it difficult, there’s no question about that,” Miller said when asked about Rangel’s ability to be effective as chairman in an interview on C-SPAN’s “Newsmakers.” “He and the members of his committee and the leadership will have to discuss that. … We have to review that. I think it’s important that the leadership review that. It’s not about dodging the issue.”

Miller is a top lieutenant of Speaker Nancy Pelosi (D-Calif.) and his comments came as Pelosi herself asked that lawmakers allow the ethics committee, formally known as the Committee on Standards of Official Conduct, to complete its inquiry into Rangel’s finances before deciding whether he should forfeit his gavel.

Miller echoed that point, saying Rangel should be given time to get a verdict on the broader investigation. “He’s entitled to have that investigation completed,” Miller said. “That may make me uncomfortable, or other Members uncomfortable, but at some point, and I realize this is a political system, so it’s not the perfect justice system, but you have to kind of do the best you can within those confines.”

But some rank-and-file Democrats were already calling Friday for Rangel’s ouster. Two others, freshman Reps. Jim Himes (Conn.) and Ann Kirkpatrick (Ariz.), confirmed Friday that they will return campaign contributions that they have received from Rangel.

“I admire Mr. Rangel for his decades of leadership on civil rights and his service in the armed forces and in Congress. But as elected officials, we must live up to the highest ethical standards,” Himes said in a statement announcing the decision.

House Democratic leadership aides, meanwhile, started talking about how to parry an expected privileged resolution from Republicans demanding the New York Democrat surrender his chairmanship.

The House last voted on such a measure back in October, after Rangel amended his personal financial disclosure forms to include more than $500,000 in assets that he had previously failed to report. The vote marked the first time since House Republicans started forcing lawmakers to go on the record over Rangel’s gavel that any Democrats — Mississippi Reps. Gene Taylor and now-Republican Parker Griffith (Ala.) in the October vote — broke ranks to support the call for him to step aside. In part to ensure more didn’t join them, Democratic leaders moved to refer the measure to the ethics committee, instead of simply tabling it as they had done on previous resolutions aimed at Rangel.

The aim if Republicans try again is to perform similar procedural jujitsu. “You want to give people something to vote for,” without actually punishing Rangel further, one senior Democratic aide said.

Miller acknowledged that it would “probably” be easier politically for Democrats if Rangel stepped down, but he said that it’s not that simple. “Sometimes in politics you have to do difficult things. I can make a case that he should step down, but I can also make a case that he’s entitled, as we say in America, to his day in court.”

And other Democrats called the ethics committee’s decision unfair to Rangel, or too minor a wrist-slap to merit his removal as the tax-writing chairman. “I think people are fair,” said Rep. Chris Van Hollen (Md.), who is charged with looking out for vulnerable Democrats as Assistant to the Speaker and Democratic Congressional Campaign Committee chairman. “This one, clearly, they found no knowing violation and there was a bipartisan decision. They didn’t recommend any further action beyond what he has now said he will do, which is pay for the trip.”