Congressional Black Caucus Chairman Rep. Cedric Richmond called the downfall of Rep. John Conyers one of the most difficult moments in Richmond’s time leading the caucus.

Conyers announced his resignation effective immediately on Tuesday amid reports he sexually harassed multiple women. The Michigan Democrat also said he endorsed his son to run for his seat.

Richmond told the New Orleans Advocate that it caused a lot of pain for members of the caucus to see one of its founding members face accusations.

“Within the family, this was the most trying moment,” he said. “A guy that you’ve known, a person you’ve grown to love and accept as part of your family, you want to hold out hope that he did not do these awful things and treat people like this.”

Watch: Jackson Lee Reads Conyers’ Resignation Statement

But Richmond said he looked at the mounting allegations, including one from BuzzFeed News that Conyers’ office paid $27,000 to settle with a woman, that seemed credible.

“However, you have no way of independently judging anything so you have to rely on the institution in (the Ethics Committee) to be that,” he said.

Richmond also said it was difficult for members of the caucus to see a leader face serious allegations.

“I could see the pain on their faces,” the Louisiana Democrat said.

But some members of the caucus like Ohio Rep. Marcia Fudge saw a double standard that led people to ask Conyers to resign while white politicians like Texas Rep. Blake Farenthold, Sen. Al Franken and President Donald Trump have not faced similar calls.

“The same double standard exists that our president uses for black people,” she said. “We are guilty until proven innocent. Without giving him due process, without even hearing what happened.”