Defenders of the proposed Ground Zero mosque are quick to write off opponents as bigots simply out to stir up hatred of Muslims.

While some of the rhetoric is excessive, the blanket dismissal hides a more difficult challenge for mosque advocates: A growing number of Muslims openly oppose the project too. Some agree it's the wrong place, while others are increasingly critical of what they see as tone-deaf and clumsy leadership from mosque planners.

The most recent, and perhaps most pointed, criticism comes from Mansoor Ijaz, who negotiated Sudan's offer of counterterrorism assistance to the FBI and CIA under the Clinton administration in 1996 and 1997. In a Washington Post column Wednesday, Ijaz faulted both sides of the debate for its tenor. But he admonished Muslims to show their sensitivity toward the country's collective emotional wound from 9/11. And, he argued, they have done more to cast themselves as victims instead of joining forces to combat the radical Islamist ideology fueling terrorism:

"[The] Cordoba House is wrong because America's Muslims do not yet exemplify the time-honored commandments, philosophies and tenets of the great men and women who founded our country -- and even more sadly, of the great religion they claim to follow. A mosque is not a monument. It is a place where worshipers gather to strengthen their beliefs en masse -- a place where they resolve to practice those beliefs with consistency and vigor. An American Muslim, one who believes in his or her American identity first, could not possibly hope to do that near the place where fellow citizens were murdered by Islamic mobsters seeking vanity and infamy for their crimes."

Even those who believe that, on principle, the project shouldn't be abandoned, acknowledge that they feel boxed in after Imam Feisal Abdul Rauf and his wife Daisy Khan sprung the $100 million mosque project on them.

Hussein Rashid makes that point in a column published Friday. Rashid, an instructor at Hofstra University's Center for Spiritual Inquiry, faulted organizers for failing to anticipate the controversy and getting caught flat-footed when it erupted. That, he argues, places Muslims in an awkward position of being "forced to defend a project about which [they are] otherwise ambivalent." To fight for the project, "we need to believe in it," and organizers' actions have built little confidence:

"This project needs a strong hand at the helm, a true visionary, but should never be dependent on that person. Rauf's noticeable absence for the last several months speaks to the problems of relying on one individual, and the overall style of leadership."

Muslims outside the country have their own concerns. Al-Arabiya television general manager Abdul Rahman Al-Rashid called the mosque "the wrong battle," saying Muslims don't want "a mosque next to the 11 September cemetery." He echoes the concern of many American opponents of the mosque, that it will be seen as a memorial to the 9/11 terrorists who acted in the name of Islam:

"The fact is that building a mosque next to the site of the World Trade Center Twin Towers, which were destroyed during the 11 September attacks, is a strange story. This is because the mosque is not an issue for Muslims, and they have not heard of it until the shouting became loud between the supporters and the objectors, which is mostly an argument between non-Muslim US citizens!

Neither did the Muslims ask for a single building, nor do the angry Muslims want the mosque. This is one of the few times when the two opposing sides are in agreement. Nevertheless, the dispute has escalated, and has reached the front pages of the press and the major television programs, demonstrations have been staged in the streets, and large posters have been hung on buses roaming the streets of New York calling for preventing the building of the mosque and reminding the people of the 11 September crime. It really is a strange battle!"

On that point, Abdur-Rahman Mohamed, a Muslim who renounced his previous radical preaching, blames organizers for triggering the backlash:

"Let us remember that the project organizers themselves created this controversy by announcing that the groundbreaking would take place on the ten-year anniversary of the attack, and that the exact site was selected because of its proximity to Ground Zero. Given that fact, the current media meme that this is not a 'Ground Zero mosque' is dishonest spin."

The Muslim Canadian Congress wrote to Rauf directly urging him to reconsider the mosque's location:

"Even if the intention of the mosque organizers was noble," the letter said, "the negative reaction by the American people has ensured that if built, the mosque will remain a permanent sore point and a lightning rod for anti-Muslim feelings. For this reason alone, the project must be abandoned."

"Many Muslims suspect that the idea behind the Ground Zero mosque is meant to be a deliberate provocation, to thumb our noses at the 'infidel,' said board member Raheel Raza. "We believe the proposal has been made in bad faith and, in Islamic parlance, is creating 'fitna,' meaning 'mischief-making,' an act clearly forbidden in the Qur'an."

Even the first Muslim Miss America found the location misguided. "We should be more concerned with the tragedy than religion," Rima Fakih said.

Neda Bolourchi lost her mother on the flight that crashed into the Trade Center's North Tower. A secular Muslim from Iran, Bolourchi is among those who see the mosque proposal as just too close to the place that put a hole in her heart. "I fear it would become a symbol of victory for militant Muslims around the world," she wrote in a Washington Post column.

M. Zuhdi Jasser, president of the American Islamic Forum for Democracy, also found fault with the proposal's lavish scale. Mosques, he wrote, are normally humble places for worship, and they shouldn't be about making statements:

"Religious freedom is central to our nation - and that is why the location of this project is so misguided. Ground Zero is purely about being American. It can never be about being Muslim."

In an article in Al-Ayyam, a Palestinian newspaper, Cambridge University researcher Khaled Al-Haroub argued the Ground Zero mosque was a needless provocation. Even it is built, it will have been "a losing battle, politically and culturally." The issue is not whether the proponents have a right to build, but whether the mosque creates more harm than good:

"[T]he principle that should guide those who wish to build the mosque transcends the [dry letter of the] law and the [goal] of scoring legal points. After all, a legal verdict permitting them to build a mosque against the will of over half the [city's] population cannot be regarded as a victory. Building a mosque near that place [i.e., Ground Zero], legally or illegally, is a provocation that hurts people's feelings, so it is inappropriate to insist on building it on that spot, of all places."

Opposition to the mosque proposal comes in different forms and from many different voices. The fact that it is not uniform indicates that there's more going on here than rank bigotry.

Editor's note: This article has been edited since its original posting to correct an error.