There are at least two lessons to be learned from the “Duck Dynasty” debacle, in which reality TV star Phil Robertson got indefinitely suspended from the A&E hit show after making anti-gay remarks in a GQ magazine interview. One: on freedom of speech, hypocrisy and double standards are rampant across the political spectrum (the title of 1992 book by the great civil libertarian Nat Hentoff, “Free Speech for Me But Not for Thee,” remains ever-relevant). Two: while some speech will always be regarded as beyond the pale in even the freest society, the rapid shifting of those boundaries is sure to generate intense cultural anxiety and conflict.

The banishment of Robertson, the patriarch of a Louisiana family that grew rich selling duck whistles, has become the latest cause of outrage on the right. Sen. Ted Cruz, the Texas Republican and likely presidential candidate, wrote on Facebook, “If you believe in free speech or religious liberty, you should be deeply dismayed over the treatment of Phil Robertson.” Sarah Palin chimed in with a tweet about our endangered freedoms. Facebook pages and petitions in support of Robertson have sprung up, along with a #StandWithPhil Twitter movement. Fanning the flames from the left, CNN’s Piers Morgan took to Twitter to assert that “the 1st Amendment shouldn’t protect vile bigots.”

It does, of course. But, as a number of commentators (including conservatives) have pointed out, the First Amendment is irrelevant to the Duck Dynasty imbroglio for a very different reason. While constitutional protections for speech certainly extend to bigots, they protect only against government actions, not sanctions by employers. There is no inalienable right to be on A&E. This isn’t even a matter of government pressure on private institutions to punish objectionable ideas—as with the 1950s Hollywood blacklists of film industry figures with Communist ties, or modern-day college speech codes targeting broadly defined “harassment.”

Still, just because A&E’s actions do not violate First Amendment freedoms doesn’t make them right. Free speech advocate Adam Kissel, former vice president of the Foundation for Individual Rights in Education (FIRE)—who has criticized Robertson’s suspension on Twitter—told me, “A&E generally has the legal right to determine the border of its tolerance and intolerance, but this does not end the moral conversation.” Kissel believes that if a private employer’s action “serves to shut out a voice that otherwise would have been heard, this choice is morally suspect.” Reason magazine senior editor Brian Doherty makes a similar argument: while censuring unpopular speech through social ostracism and economic boycott may not be un-libertarian, it’s deeply illiberal and contrary to the spirit of tolerance that makes society flourish.

Libertarians who make this argument have ideological consistency on their side. (FIRE has staunchly opposed all curbs on “offensive” speech on college campuses, whether the offense was to feminist and multicultural sensitivities or Christian and patriotic ones.) Conservatives, on the other hand, have their own long record of trying to silence or punish expression they dislike—including pro-gay expression.

Take the American Family Association, which charges that A&E “believes in freedom of speech…only if it is the speech content with which they agree.” But so does the AFA: boycotts directed against “immoral”—and, specifically, gay-friendly—content have been practically its bread and butter. In 1997, the group fought to stop ABC from having the lead on “Ellen” come out as a lesbian. L. Brent Bozell of the Media Research Center, another vocal critic of the alleged Duck Dynasty persecution, supported both the “Ellen” boycott (the MRC took out a full-page ad in Variety condemning the “blatant attempt by Disney, ABC and ‘Ellen’ to promote homosexuality to America's families”) and that of “Nothing Sacred,” a 1997-1998 ABC show that angered conservative Catholics by portraying a priest who struggled with his faith and questioned Church teachings on sexuality.

Backlash from the right has targeted other kinds of speech, too. Ten years ago, there was the Dixie Chicks boycott, which more or less destroyed the country music band after lead singer Natalie Maine opened a London concert by saying that they opposed the war in Iraq and were “ashamed” to be from the same state as President Bush. (Even sympathetic programmers had to take Dixie Chicks songs off the air, and some boycott-defying DJs lost their jobs.) Back then, conservatives—including President Bush—were the ones pointing out that while the Dixie Chicks were free to take a stand, so were their detractors.

And, just last year, Miami Marlins manager Ozzie Guillen got in trouble by gushing to Time magazine that he loved and respected Cuba’s retired dictator Fidel Castro for his longevity in the face of so many enemies. Anti-Communist Cuban-American groups threatened a boycott, demanding Guillen’s resignation. He was suspended for five games and made groveling apologies; a few months later he was fired, almost certainly due in part to the Castro flap.

To some extent, ostracism, no less than speech, is the exercise of a constitutional right: freedom of association. And some opinions are ostracized by near-universal agreement. If an entertainer had praised Hitler or Osama Bin Laden, defended pedophilia, or endorsed wife-beating, his career would be over faster than you can say “First Amendment” (and no one would be clamoring for his freedom of speech). Holocaust denial, 9/11 “trutherism,” and claims about the genetic inferiority of some racial and ethnic groups have all been effectively driven to the fringes of the marketplace of ideas—where most of us would prefer them to stay.

The tough question is where to draw the lines when social standards shift. In two generations, near-universal harsh disapproval of homosexuality has given way to harsh disapproval of homophobia. Interestingly, while traditional views of male and female roles are widely regarded as outmoded, they are generally not equated with outright bigotry; traditional views of homosexuality, however, seem headed in that direction.

Granted, Phil Robertson did more than express the biblical view of homosexual conduct as sin: his ramblings seemed to lump together homosexuality and bestiality (though he put heterosexual promiscuity on the same list) and included bizarre anatomically correct remarks about the superior joys of heterosexual sex. But, these particulars aside, there is an unmistakable trend toward marginalizing all opinion that treats same-sex relationships less favorably than male-female unions—whether on religious grounds or because of the latter’s reproductive potential and sexual complementarity—as we have marginalized bias against interracial marriage. Will that really benefit public discourse, or worsen cultural tensions because of apparently well-founded fears that gay equality leads to intolerance toward traditionalists?

Those fears are likely to be exacerbated by the fact that societal standards are still in flux. In recent months, for instance, there has been a noticeable increase in commentary suggesting that acceptance of non-monogamous relationships should be the next social frontier. It may not be entirely paranoid for conservatives to wonder if, eventually, “enlightened” society will seek to banish anyone voicing an unflattering opinion of polyamory.

In a decentralized economy with unprecedented media diversity, ostracism by the mainstream is unlikely to silence dissenters or strip them of their livelihood. (“Duck Dynasty” has already received offers for a show if its relationship with A&E is terminated.) Increased fragmentation and polarization in the marketplace of ideas is a far likelier consequence—and not a good one. In that sense, Doherty is right to say that the power of exclusion should be used very sparingly.

What does this mean for A&E and “Duck Dynasty”? Frankly, I don’t think Phil Robertson—whose interview also featured cartoonish reminiscences about “godly” blacks singing happy songs in the Jim Crow-era South—is someone conservatives should “stand with.” But, like other commentators left and right, I agree that A&E’s response was ludicrously excessive, particularly since Robertson’s remarks fit perfectly with his reality TV image on which the network has capitalized. The smartest thing A&E could do is wait out the controversy and then bring Robertson back once the show resumes filming; of course, that may have been the plan all along.

By then, the rest of us will have moved on to new paroxysms of outrage at offensive speech. Maybe next time, a reality-show gay dad can offend by mocking evangelical Christians as deluded idiots. Then, conservatives can call for his head and denounce anti-Christian bigotry, and liberals can stand up for his right to free speech.