House Ways and Means Chairman Charlie Rangel (D-N.Y.) is meeting with Speaker Nancy Pelosi (D-Calif.) and other Democratic leaders amid questions about his continued leadership of the tax-writing panel.

Rangel walked in Pelosi’s office about 5 p.m. and brushed off reporters’ questions. Also in the meeting are Majority Leader Steny Hoyer (D-Md.) and Democratic Caucus Vice Chairman John Larson (Conn.).

Rangel has been under fire for his personal finances, and some groups have called for him to step aside until the ethics committee rules.