Why has the Supreme Court held that felons lack the constitutional right to vote?

Occasionally, people ask about the constitutional basis for denying felons the right to vote (set aside here the policy arguments about that). The answer is that it's there in the little-known section 2 of the Fourteenth Amendment:

But when the right to vote at any election for the choice of electors for President and Vice President of the United States, Representatives in Congress, the Executive and Judicial officers of a State, or the members of the Legislature thereof, is denied to any of the male inhabitants of such State, being twenty-one years of age, and citizens of the United States, or in any way abridged, except for participation in rebellion, or other crime, the basis of representation therein shall be reduced in the proportion which the number of such male citizens shall bear to the whole number of male citizens twenty-one years of age in such State.

Given that it is section 1 of the Fourteenth Amendment that has been read as generally securing a constitutional right to vote, I think that right has to be read in light of the restrictions that section 2 says are tolerable. And that is precisely what the Supreme Court held in Richardson v. Ramirez (1974) (some paragraph breaks added):

Despite this settled historical and judicial understanding of the Fourteenth Amendment's effect on state laws disenfranchising convicted felons, respondents argue that our recent decisions invalidating other state-imposed restrictions on the franchise as violative of the Equal Protection Clause require us to invalidate the disenfranchisement of felons as well….

As we have seen, however, the exclusion of felons from the vote has an affirmative sanction in § 2 of the Fourteenth Amendment, a sanction which was not present in the case of the other restrictions on the franchise which were invalidated in the cases on which respondents rely. We hold that the understanding of those who adopted the Fourteenth Amendment, as reflected in the express language of § 2 and in the historical and judicial interpretation of the Amendment's applicability to state laws disenfranchising felons, is of controlling significance in distinguishing such laws from those other state limitations on the franchise which have been held invalid under the Equal Protection Clause by this Court….

[We] rest on the demonstrably sound proposition that § 1, in dealing with voting rights as it does, could not have been meant to bar outright a form of disenfranchisement which was expressly exempted from the less drastic sanction of reduced representation which § 2 imposed for other forms of disenfranchisement. Nor can we accept respondents' argument that because § 2 was made part of the Amendment "'largely through the accident of political exigency rather than through the relation which it bore to the other sections of the Amendment,'" we must not look to it for guidance in interpreting § 1. It is as much a part of the Amendment as any of the other sections, and how it became a part of the Amendment is less important than what it says and what it means.

Pressed upon us by the respondents, and by amici curiae, are contentions that these notions are outmoded, and that the more modern view is that it is essential to the process of rehabilitating the ex-felon that he be returned to his role in society as a fully participating citizen when he has completed the serving of his term. We would by no means discount these arguments if addressed to the legislative forum which may properly weigh and balance them against those advanced in support of California's present constitutional provisions.

But it is not for us to choose one set of values over the other. If respondents are correct, and the view which they advocate is indeed the more enlightened and sensible one, presumably the people of the State of California will ultimately come around to that view. And if they do not do so, their failure is some evidence, at least, of the fact that there are two sides to the argument.

Some readers might ask whether section 2's implicit endorsement of the constitutionality of requirements that voters be "male" and "twenty-one years of age" would similarly render constitutional restrictions on voting by women and by 18-to-20-year-olds. The answer, I think, is surely yes—which is why it took the Nineteenth Amendment and the Twenty-Sixth Amendment to specifically forbid such voting restrictions. (Note that voting restrictions may also be unconstitutional if they are motivated by a desire to discriminate based on race, see Hunter v. Underwood (1985). But many laws barring felons from voting were motivated by hostility to felons generally, not hostility to a particular racial group.)

Finally, if you want to be really precise, the Fourteenth Amendment hasn't quite been read as securing a "right to vote" the way that it has been read as securing a right to free speech, a right to send your children to private school, or a right to abortion. Indeed, governments can often entirely abolish the right to vote, at least for political offices, by simply making them appointed offices. (A state, for instance, can switch from electing judges to having the governor appoint them, without violating anyone's right to vote.)

Rather, the "right to vote" under the U.S. Constitution is an equality right—a right not to be discriminated against based on, for instance, whether you pay property taxes in the district, or whether you have children who go to public schools. Section 2 of the Fourteenth Amendment, though, makes clear that some kinds of discrimination, such as based on criminal convictions, citizenship, and residence, are indeed permissible. And state constitutions, of course, often do explicitly secure rights to vote, because they do specifically provide what offices are elective offices and who may vote for those offices.

(Special bonus: Now that states have generally repealed fornication laws, the right to marry is likely in the same position as the right to vote. It's a right to be treated equally when it comes to marriage laws, for instance not to be denied the benefits of marriage because you have outstanding child support obligations. But it's not a pure liberty right: The government could abolish all state recognition of marriage—not that it's likely to—without violating anyone's "right to marry." In practice, these rights are called the "right to vote" and the "right to marry," but conceptually they are equality rights rather than liberty rights such as free speech or the freedom from unreasonable searches and seizures.)