House Judiciary Committee leaders are threatening to impeach former U.S. District Judge Samuel B. Kent if he doesn’t resign immediately.

Chairman John Conyers (D-Mich.) and ranking member Lamar Smith (R-Texas.) on Monday warned Kent, a Texas judge sentenced to nearly three years in jail for sexually abusing a court staffer, that he had better step down now if he doesn’t want to be forced out.

“Unless Judge Samuel Kent immediately resigns, we intend to introduce a resolution jointly [Tuesday] to commence an inquiry into whether grounds exist to impeach him and remove him from office,— Conyers and Smith said in a joint statement.

In February, as part of his guilty plea, Kent agreed to resign from the federal bench. He hasn’t done so and has since filed for disability in order to receive his full pension, which is a lifetime salary equivalent to his current pay.