While negotiators met at the office of the U.S. Trade Representative late last week in their latest bid to thrash out the many details in the Trans-Pacific Partnership, a boisterous collection of union organizers and other activists made clear there is only one detail that matters to them: jobs.

It was a loud and colorful display of the difficulty the Obama administration faces in selling one of its most important constituencies on one of its stated trade goals in the president’s second term.

Labor unions may have helped elect President Barack Obama, but questions over international trade pit many in the movement against the president. Many are lining up against the ambitious TPP negotiations that are the centerpiece of the administration’s trade goals, and they’re looking to block the fast-track trade authority that the White House needs to advance any trade plan through Congress.

“No TPP and no fast track,” the protesters chanted from the street as international officials met inside.

The Obama administration has said it wants to conclude the ambitious Pacific trade pact by year’s end, and it has urged Congress to grant Obama fast-track trade authority to help ease the pact’s passage through the legislative branch.

Unions point to a record they say shows that free-trade agreements have led to U.S. job losses and wage stagnation, and even those labor leaders not protesting TPP and other trade initiatives have been pressuring the administration to do more on labor issues in trade deals.

“We’re not against trade, we’re just against trade that keeps costing us jobs,” said Leo Gerard, international president of the United Steelworkers, who was not at the labor rally.

Gerard said that since the North American Free Trade Agreement took effect in 1994, the U.S. manufacturing sector has taken a beating. Such pacts, he said, “are investment deals that allow multinational corporations to play labor arbitrage.”

The administration bristles at such charges. U.S. Trade Representative Michael Froman says the president and his aides have always made it clear any new trade deals must meet high standards, especially for workers and their wages, promoting jobs and serving as a vehicle to strengthen the middle class.

“We have engaged extensively with the labor movement,” said Froman, who won Senate confirmation in June. “If you look across everything we’re doing, there is a long arc toward greater labor protections in trade agreements.”

Labor groups are responding to a yearslong shift of jobs from the United States, a movement labor leaders argue is being pushed along by trade pacts that allow companies to use cheaper labor to source goods abroad. The Commerce Department, for instance, said in a 2011 report that large multinational companies cut 2.9 million jobs in the U.S. in the 2000s while adding 2.4 million jobs overseas.

Such reports provide a hurdle to the administration’s efforts to win over trade skeptics in the labor movement.

Gerard said his union opposes granting Obama fast-track authority, a procedural action that is considered necessary to pass any trade agreement because it gives Congress only an up-or-down vote on a signed deal. And officials with other unions also appear to be leaning that way.

Mike Dolan, a legislative representative for the International Brotherhood of Teamsters who focuses on trade policy, said his union hadn’t yet made up its mind on the TPP deal. But protesters wearing Teamsters T-shirts participated in last week’s rally at the USTR office, and Dolan said he believes the closed-door negotiations do not look to be trending labor’s way.

“The TPP is within a tradition of the last 20 years of what we feel is a flawed and failed model for so-called free trade,” Dolan said. “The basis from labor’s perspective is the sheer volume of jobs that have been lost under these agreements, beginning with NAFTA.”

Celeste Drake, trade policy specialist with the AFL-CIO, said her union has not taken an official position on TPP or fast track but isn’t overly optimistic that either will include the “reforms” her side is urging.

Still, the AFL-CIO isn’t giving up, she said.

Drake said she and other staffers remain in contact with the USTR and congressional aides and that AFL-CIO President Richard Trumka “does speak to Ambassador Froman personally about trade issues.”

At last week’s demonstration, Kenneth Peres, chief economist for the Communications Workers of America, an AFL-CIO affiliate, spoke out against the TPP and the labor conditions in countries participating in the talks. “Remember, they want to put our workers in competition with countries like Vietnam, who have a minimum wage of $2.23, that’s not an hour … for a full day’s work,” he said.

Unions are interested not only in influencing the labor chapter of the sweeping TPP deal but in some of the other 29 chapters involving investment rules, country-of-origin standards for product labeling and other controversial matters.

If the AFL-CIO ends up opposing a TPP deal — if such an agreement is signed and sent to Congress for approval — Drake said it will mount a serious campaign against the pact. “At a time when we’re really trying to grow and see a resurgence of a strong American labor movement, that is not the time I see the affiliates laying down and saying we’ll accept a job-killing trade agreement,” she said.

Proponents of the TPP and of fast track, such as Calman Cohen, president of the Emergency Committee for American Trade, argue that without new trade deals such as the TPP, Americans may lose existing and potential jobs.

“We need trade agreements to keep markets open overseas,” Cohen said. “If we don’t, the [U.S.] products, the manufactured products … will not have those who can consume or use them. And that means the jobs that are related to the production of those products are in jeopardy.”

Administration officials argue, too, that international commerce, particularly in Asia, will go on whether the TPP deals goes through or not. But, they say, the United States can use the deal to nudge some of its trading partners to do more to protect workers’ rights abroad and to help level the field for American workers to compete.

The Obama administration notes it has pursued workers’ rights through such actions as suspending the generalized system of preferences trade benefits for Bangladesh. And it brought an extremely rare trade case using a labor provision against Guatemala. “This administration has demonstrated its commitment to addressing labor concerns in trade agreements,” Froman said. “And it has a proven record of doing so.”

Drake lauds the administration on some aspects of trade policy, such as a World Trade Organization case against Chinese tire imports.

“In a couple places I think there’s measurable progress that we can point to,” she said. “This administration has been just head and shoulders above the prior administration in terms of pursuing trade cases particularly with respect to China.”

But Froman indicated he knows he has a tough audience when it comes to labor and its Democratic allies on Capitol Hill.

“We need to work with members of both parties, with business, with labor, with [non-governmental organizations] in our goal to build a broad base of support for trade agreements,” Froman said.