For many libertarians, "the road to serfdom" is not just the title of a great book but also the window through which they see the world. We’re losing our freedom, year after year, they think. They (we) quote Thomas Jefferson: “The natural progress of things is for liberty to yield and government to gain ground.” We read books with titles like Freedom in Chains, Lost Rights, The Rise of Federal Control over the Lives of Ordinary Americans, and yes, The Road to Serfdom.

The Cato Institute's boilerplate description of itself used to include the line, "Since [the American] revolution, civil and economic liberties have been eroded." Until Clarence Thomas, then chairman of the Equal Employment Opportunity Commission, gave a speech at Cato and pointed out to us that it didn't seem quite that way to black people.

And he was right. American public policy has changed in many ways since the American Revolution, sometimes in a libertarian direction, sometimes not.

Brink Lindsey talks of an "implicit libertarian synthesis" in American politics today in his book The Age of Abundance. He argued in 2007:

Nevertheless, the fact is that American society today is considerably more libertarian than it was a generation or two ago. Compare conditions now to how they were at the outset of the 1960s. Official governmental discrimination against blacks no longer exists. Censorship has beaten a wholesale retreat. The rights of the accused enjoy much better protection. Abortion, birth control, interracial marriage, and gay sex are legal. Divorce laws have been liberalized and rape laws strengthened. Pervasive price and entry controls in the transportation, energy, communications, and financial sectors are gone. Top income tax rates have been slashed. The pretensions of macroeconomic fine-tuning have been abandoned. Barriers to international trade are much lower. Unionization of the private sector work force has collapsed. Of course there are obvious counterexamples, but on the whole it seems clear that cultural expression, personal lifestyle choices, entrepreneurship, and the play of market forces all now enjoy much wider freedom of maneuver.

Has there ever been a golden age of liberty? No, and there never will be. There will always be people who want to live their lives in peace, and there will always be people who want to exploit them or impose their own ideas on others. If we look at the long term—from a past that includes despotism, feudalism, absolutism, fascism, and communism—we’re clearly better off. When we look at our own country's history—contrasting 2010 with 1776 or 1910 or 1950 or whatever—the story is less clear. We suffer under a lot of regulations and restrictions that our ancestors didn’t face.

But in 1776 black Americans were held in chattel slavery, and married women had no legal existence except as agents of their husbands. In 1910 and even 1950, blacks still suffered under the legal bonds of Jim Crow—and we all faced confiscatory tax rates throughout the postwar period.

I am particularly struck by libertarians and conservatives who celebrate the freedom of early America, and deplore our decline from those halcyon days, without bothering to mention the existence of slavery. Take R. Emmett Tyrrell, Jr., longtime editor of the American Spectator. In Policy Review (Summer 1987, not online), he wrote:

Let us flee to a favored utopia. For me that would be the late 18th Century but with air conditioning....With both feet firmly planted on the soil of my American domain, and young American flag fluttering above, tobacco in the field, I would relish the freedom.

I take it Mr. Tyrrell dreams of being a slave-owner. Because as he certainly knows, most of the people in those tobacco fields were slaves.

Take a more recent example, from a libertarian. Jacob Hornberger of the Future of Freedom Foundation writes about the decline of freedom in America:

First of all, let’s talk about the economic system that existed in the United States from the inception of the nation to the latter part of the 19th century. The principles are simple to enumerate: No income taxation (except during the Civil War), Social Security, Medicare, Medicaid, welfare, economic regulations, licensure laws, drug laws, immigration controls, or coercive transfer programs, such as farm subsidies and education grants.

There was no federal department of labor, agriculture, commerce, education, energy, health and human services, or homeland security.

Then he writes:

Why did early Americans consider themselves free? The answer is rooted in the principles enunciated in the Declaration of Independence. As Thomas Jefferson observed in that document, people have been endowed by their Creator with certain fundamental and inherent rights. These include, but are certainly not limited to, the rights to life, liberty, and the pursuit of happiness.

But wait. Did "early Americans consider themselves free"? White Americans probably did. But what about black Americans, and especially the 90 percent of black Americans who were slaves? Slaves made up about 19 percent of the American population from 1790 to 1810, dropping to 14 percent by 1860. (In that period the number of slaves grew from 700,000 to about 4 million, but the rest of the population was growing even more rapidly.) Did Mr. Hornberger really forget that 4 million Americans were held in bondage when he waxed eloquent about how free America was until the late 19th century? I know he isn't indifferent to the crime of slavery. But too many of us who extol the Founders and deplore the growth of the American state forget that that state held millions of people in chains. (I note that I'm not concerned here with self-proclaimed libertarians who join neo-Confederate organizations or claim that southerners established a new country and fought a devastating war for some reason other than the slavery on which their social and economic system rested; I just want to address libertarians who hate slavery but seem to overlook its magnitude in their historical analysis.)

If you had to choose, would you rather live in a country with a department of labor and even an income tax or a Dred Scott decision and a Fugitive Slave Act?

I said that white Americans probably considered themselves free. But in retrospect, were they? They did not actually live in a free society. They were restricted in the relations they could have with millions of their—I started to say "their fellow citizens," but of course slaves weren't citizens—their neighbors. They lived under a despotic power. Liberalism seeks not just to liberate this or that person, but to create a rule of law exemplifying equal freedom. By that standard, even the plantation owners did not live in a free society, nor even did people in the "free" states.

Hornberger goes on:

A critical question arises: What do the rights to life, liberty, and the pursuit of happiness connote?
For our American ancestors, such rights meant more than the absence of physical constraint, e.g., not being incarcerated in jail.

Freedom also meant the right to criticize government officials and protest their actions without being punished for it.
It meant the right to worship, each in his own way, or, on the other hand, not to worship at all.

It meant the right to keep and bear arms, not only as a protection against criminals and invaders but also to ensure that the right to resist tyranny was retained by the people.

It meant the protection of centuries-old procedures in the event of federal criminal prosecutions, including habeas corpus, right to counsel, trial by jury, bail, due process of law, and protection from coerced confessions, unreasonable searches, and cruel and unusual punishments.

To our ancestors, however, freedom meant even more than that, and there arises the rub with today’s liberals. Freedom, our ancestors maintained, also meant the right to keep everything you owned and to decide for yourself what to do with it. Everyone had the right, they contended, to pursue an occupation or trade without seeking the permission of the government. They had the right to enter into mutually beneficial trades with others who were doing the same thing.

And again I say, when he says "our American ancestors," he's thinking only of our white ancestors. Maybe only of our white male ancestors. Maybe even only of our white male property-owning ancestors. Many millions of Americans would read these paragraphs and say, "My ancestors didn't have the right to worship in their own way. My ancestors didn't have the right to keep and bear arms. My ancestors didn't have the protection of centuries-old legal procedures. My ancestors sure as heck didn't have the right to keep what they produced, or to pursue an occupation of their choice, or to enter into mutually beneficial trades. In fact, my ancestors didn't even have the minimal right of 'the absence of physical constraint.'"

I've probably been guilty of similar thoughtless and ahistorical exhortations of our glorious libertarian past. And I'm entirely in sympathy with Hornberger's preference for a world without an alphabet soup of federal agencies, transfer programs, drug laws, and so on. But I think this historical perspective is wrong. No doubt one of the reasons that libertarians haven't persuaded as many people as we'd like is that a lot of Americans don't think we're on the road to serfdom, don't feel that we've lost all our freedoms. And in particular, if we want to attract people who are not straight white men to the libertarian cause, we'd better stop talking as if we think the straight white male perspective is the only one that matters. For the past 70 years or so conservatives have opposed the demands for equal respect and equal rights by Jews, blacks, women, and gay people. Libertarians have not opposed those appeals for freedom, but too often we (or our forebears) paid too little attention to them. And one of the ways we do that is by saying "Americans used to be free, but now we're not"—which is a historical argument that doesn't ring true to an awful lot of Jewish, black, female, and gay Americans.

But it's not just a strategic mistake. It's a mistake. Whether we were more free at some point in the past than we are now is a complicated issue. I would tend to argue that we were not. But at least it's a difficult issue.

Hornberger lists a lot of federal agencies and programs that didn't exist in the 19th century. But that doesn't mean that the era was a libertarian paradise. As Jonathan R. T. Hughes wrote in The Governmental Habit Redux, “Most studies of modern nonmarket controls consider that the relevant history extends back to the New Deal. A few go back further, into the late nineteenth century. But in fact the powerful and continuous habit of nonmarket control in our economy reaches back for centuries. Thus, during the colonial period virtually every aspect of economic life was subject to nonmarket controls. Some of this tradition would not survive, some would become even more powerful, while some would ascend to the level of federal control. The colonial background was like an institutional gene pool. Most of the colonial institutions and practices live on today in some form, and there is very little in the way of nonmarket control that does not have a colonial or English forerunner.”

The dramatic issue of slavery reminds us that the level of taxes or the number of federal agencies is by no means the only measure of freedom. And we can imagine other examples where the common libertarian focus on economic issues could lead us astray.

Washington Post columnist E. J. Dionne, Jr. offers this challenge to advocates of “smaller government”: Imagine a choice between “a dictatorship in which the government provides no social security, health, welfare, or pension programs of any kind” and “levies relatively low taxes that go almost entirely toward the support of large military and secret police forces that regularly kill or jail people for their political or religious views” and “a democracy with open elections and full freedom of speech and religion [which] levies higher taxes than the dictatorship to support an extensive welfare state.” “The first country might technically have a ‘smaller government,’” Dionne writes, “but it undoubtedly is not a free society. The second country would have a ‘bigger government,’ but it is indeed a free society.”

Now there are several problems with this comparison, not least Dionne’s apparent view that high taxes don’t limit the freedom of those forced to pay them. And the rarity in the real world of dictatorships with secret police forces that have low taxes. But let’s just look at which one might be called the “smaller government.” Measured as a percentage of GDP or by the number of employees, the second government may well be larger than the first. Measured by its power and control over individuals and society, however, the first government is doubtless larger. Libertarians want a government that is limited in size, scope, and power.

We often focus on the size of government, as measured in percentage of GDP taxed and spent by the government, which is an important and measurable concept. But our real concern is power. What kind of power does the government wield over the people? Powerful state institutions tend to be large, but that doesn't mean that a larger state is necessarily exercising more power. Imagine a small town that adds two officers to its police force. Now it has more police officers, and that costs more money; the government is "larger." But if the officers now do a better job of arresting violent criminals and protecting the lives and property of the people—and refrain from arresting or hassling non-criminals—then the government has not expanded its power. Indeed, better eight officers protecting lives and property than six officers enforcing drug laws and blue laws. We should focus on what is actually important—the exercise of arbitrary power over others. And in that regard slavery and conscription, among other things that marred parts of our American past, loom very large.

David Boaz is executive vice president of the Cato Institute and author of Libertarianism: A Primer and The Politics of Freedom.

Editor's Note: Read Jacob Hornberger's response to this article here.