Gifford Miller (D) has had two jobs in his adult life: Congressional aide and New York City Councilman, including his tenure as Council Speaker since January 2002.

Now term limited at the tender age of 35, Miller is running for a job that’s frequently described as the second-most difficult in America: mayor of New York. But his path to the November general election may be blocked by Rep. Anthony Weiner (D-N.Y.), the other white candidate in a four-way Democratic primary.

In the spicy stew of New York ethnic politics, the question cannot be avoided: In a field with a Latino man (former Bronx Borough President Fernando Ferrer, the early frontrunner) and a black woman (Manhattan Borough President C. Virginia Fields), will the two white guys cancel each other out?

To be fair, a similar question is also raised about the potential for Fields and Ferrer to hurt each other’s chances. The last time black and Latino voters seemed truly united in New York City, David Dinkins (D) was elected the city’s first black mayor — but that was 16 years ago.

Miller, a former aide to Rep. Carolyn Maloney (D-N.Y.), doesn’t see Weiner as an impediment, however.

“I don’t worry about Congressman Weiner or any particular candidate,” Miller said in an interview Wednesday — a day spent on Weiner’s turf in Washington, D.C., mixing official duties with political outreach. “I’m worried about what I’m doing each day.”

Yet it is clearly in Miller’s interest to see Weiner fade, and he is quick — almost gleeful — to point out that most of the Democratic clubs in Weiner’s Brooklyn-Queens district have spurned the Congressman in favor of endorsing Miller.

That accomplishment fits nicely with Miller’s strategy of billing himself as the candidate for all the city — not just the tony Upper East Side of Manhattan, where he grew up and which he represents on the Council.

“You have to be competing for votes in every community, and I’m not sure if any other candidate is proving able to do that,” Miller said.

The platform of Council Speaker has allowed Miller to advance that argument, because many of his Council colleagues are backing him.

New York City’s primary electorate is frequently balkanized, not just by race but also by borough and even neighborhood. Yet the latest independent poll on the primary seems to have shaken up some of the conventional wisdom.

In the borough-by-borough breakdown of a Quinnipiac University poll conducted earlier this month, only one candidate had a commanding lead in his home territory — Ferrer in the Bronx. Everywhere else, the four candidates were more or less bunched together.

Ferrer had a huge lead among Hispanic voters, and Fields ran strongest among blacks. But the white vote was evenly divided: Weiner led with 23 percent, followed by Ferrer and Miller with 17 percent each and Fields with 16 percent.

City-wide for the past several months, Weiner and Miller have brought up the rear in most public polls, with support in the low double digits. In the latest Quinnipiac poll, Ferrer led with 27 percent — a significant drop from his previous showings — while Fields had 23 percent, Weiner 13 percent and Miller 11 percent.

If no candidate wins 40 percent in the Sept. 13 primary, the top two finishers will advance to a runoff two weeks later. The winner will take on billionaire Mayor Michael Bloomberg (R), who spent more than $60 million of his own money on his election four years ago and promises more of the same this time.

Bloomberg led each of the Democrats in the latest Quinnipiac poll.

But Miller discounts the polls’ significance at this early stage.

“There’s no reason I should be way ahead in the polls right now, because nobody knows who I am,” he conceded. “I think the biggest problem getting my numbers up in the primary is getting people to know me.”

He also noted that while Bloomberg leads all Democrats, the mayor’s numbers are only in the 40s, suggesting that there are plenty of voters willing to consider someone else.

Miller has been methodical about pulling together support, and he believes he can surprise political observers, just as he did when he won the Speaker’s race three and a half years ago.

Back then, Miller boosted his ambitions by creating a political action committee that backed Democratic candidates in 38 Council races in 2001; many of the grateful winners then cast their votes for him for Speaker.

This time, Miller is once again well-positioned financially to boost his name recognition as the race goes on. He has raised about $7 million — more than any other Democrat — and has put together an experienced team of advisers.

Mandy Grunwald is handling Miller’s media and has already been paid $100,000, according to Wednesday’s New York Daily News. Mark Mellman is Miller’s pollster, and Brian Hardwick, a former national finance director for then-Vice President Al Gore as well as deputy manager of the 2004 presidential campaign of Sen. Joe Lieberman (D-Conn.), is serving as campaign manager.

Weiner, for his part, has countered with another celebrity hire, according to Wednesday’s New York Times, bringing in Michael Whouley, the Dewey Square Group partner who is a close confidant to 2004 Democratic presidential nominee Sen. John Kerry (Mass.).

Miller said he offers voters two seemingly contradictory attributes. One is his youth; the other is his experience. Due to the city’s two-term limit and the fact that he won a special Council election in early 1996, Miller is now the senior elected official in New York.

“I’m in the unique position of having a fresh face and a proven record,” he said. “Everybody talks about what they want to do. I’m the only one who has done anything.”