Speaker Paul D. Ryan insists he’s not giving up his gavel before November, despite a push from some members of his conference to avoid a protracted leadership battle. 

“I don’t think most members want me to do that,” the Wisconsin Republican said on “CBS This Morning.”

“I serve of course in this job at the pleasure of the members. But I think that kind of disruption is probably not good for us as party, good for us as House Republicans. I think we have a seamless leadership team that can take over at the right time.”

While Ryan is likely correct that most members don’t want him to relinquish the speaker gavel early, many do want the House Republican Conference to avoid a protracted leadership battle for his post. 

Several members have said they hope the issue gets settled quickly in the next few weeks behind the scenes and that an heir apparent becomes clear. 

Watch: Three Questions About Ryan’s Future

Perhaps in an effort to facilitate that, Ryan said he has a preference on who should succeed him. Although he declined to specify who that is, he noted he’ll probably have more to say on the topic “pretty soon.”

The two leading candidates to take over for Ryan are House Majority Leader Kevin McCarthy and Majority Whip Steve Scalise, the No. 2 and No. 3, respectively. Ryan is likely to endorse one of them, noting that he has a “fantastic” leadership team that can easily fill the vacuum he will leave.

“I love this entire leadership team,” he said. “I feel really strongly about it. We’ve gone through a lot together. We’re very close. And we — what I like about this particular leadership team is we all row the boat in the same direction. And that necessarily wasn’t the case with some prior teams.”

Ryan also continued to insist that President Donald Trump was not a factor in his decision to retire.

“It’s really a timing of family, and it’s the fact that I accomplished much of the agenda I came here to accomplish,” he said.

Regarding his relationship with Trump, Ryan said, “We have a very good, very candid dialogue. And I find it’s better to talk to the president than talk about the president on the TV, on media.”

Ryan added that Trump didn’t make his job difficult.

“He gave us the ability to get historic things in law that we’ve been trying to get into law for a generation,” he said. “That’s not making things difficult. That’s actually facilitating real reform.”

Trump hosted a dinner at the White House Wednesday night with Ryan and his potential successors, McCarthy and Scalise. Senate Majority Leader Mitch McConnell and Majority Whip John Cornyn were also present. 

Asked about a picture of the GOP leaders and the president from the dinner standing in a line all giving a thumbs-up, Ryan said, “We were celebrating the agenda we’ve passed and the agenda we have yet to pass in this session in this spring.”

“We’re working on infrastructure, we’re working on some health care bills, we’re working on lots of different bills with respect to workforce development, career and technical education,” he said. 

Gail King, who was conducting the “This Morning” interview, noted that the photo of all white men doesn’t make people like her — a black woman — feel included or represented. 

Ryan said he didn’t want her to feel that way and noted that he’s made an effort to recruit minority women GOP candidates to run for office, like Utah Rep. Mia Love. He said he would continue to be involved in recruitment after his retirement. 

“I’m not going away from life,” he said. “I’m going to keep being involved and focusing on inclusive, aspirational politics. I’m going to keep fighting for the things I believe in, and that’s among the things I want to do. I’m going to spend a lot of time with my friend Bob Woodson on poverty initiatives. Those are things I care a great deal about.”