Who first said, 'The best government is that which governs least'? Not Thoreau.
The phrase "that government is best which governs least" is often credited to Henry David Thoreau, in his 1849 "Civil Disobedience," or "Resistance to Civil Government." (It's also sometimes credited to Thomas Jefferson or John Locke, but although it might capture well some of their thinking, to my knowledge it doesn't appear in their writings.) But Thoreau was drawing on an existing, nearly identical phrase, "The best government is that which governs least"; and he was doing it to actually argue for outright abolition of government rather than just small government:
I heartily accept the motto,—"That government is best which governs least"; and I should like to see it acted up to more rapidly and systematically. Carried out, it finally amounts to this, which I also believe,—"That government is best which governs not at all"; and when men are prepared for it, that will be the kind of government which they will have. Government is at best but an expedient; but most governments are usually, and all governments are sometimes, inexpedient.
The motto he was referring to was apparently that of the United States Magazine and Democratic Review, founded in 1837 by John O'Sullivan (who would in 1845 famously use the phrase "manifest destiny" to support the annexation of Texas and the Oregon Country). In its first issue, on pp. 6-7, the Review included this passage (paragraph breaks and emphasis added):
It is under the word government, that the subtle danger lurks. Understood as a central consolidated power, managing and directing the various general interests of the society, all government is evil, and the parent of evil.
A strong and active democratic government, in the common sense of the term, is an evil, differing only in degree and mode of operation, and not in nature, from a strong despotism. This difference is certainly vast, yet, inasmuch as these strong governmental powers must be wielded by human agents, even as the powers of the despotism, it is, after all, only a difference in degree; and the tendency to demoralization and tyranny is the same, though the development of the evil results is much more gradual and slow in the one case than in the other. Hence the demagogue—hence the faction—hence the mob—hence the violence, licentiousness, and instability—hence the ambitious struggles of parties and their leaders for power—hence the abuses of that power by majorities and their leaders—hence the indirect oppressions of the general by partial interests—hence (fearful symptom) the demoralization of the great men of the nation, and of the nation itself, proceeding, unless checked in time by the more healthy and patriotic portion of the mind of the nation rallying itself to reform the principles and sources of the evil) gradually to that point of maturity at which relief from the tumult of moral and physical confusion is to be found only under the shelter of an energetic armed despotism.
The best government is that which governs least. No human depositories can, with safety, be trusted with the power of legislation upon the general interests of society so as to operate directly or indirectly on the industry and property of the community. Such power must be perpetually liable to the most pernicious abuse, from the natural imperfection, both in wisdom of judgment and purity of purpose, of all human legislation, exposed constantly to the pressure of partial interests; interests which, at the same time that they are essentially selfish and tyrannical, are ever vigilant, persevering, and subtle in all the arts of deception and corruption.
In fact, the whole history of human society and government may be safely appealed to, in evidence that the abuse of such power a thousand fold more than overbalances its beneficial use. Legislation has been the fruitful parent of nine-tenths of all the evil, moral and physical, by which mankind has been afflicted since the creation of the world, and by which human nature has been self-degraded, fettered, and oppressed.
Government should have as little as possible to do with the general business and interests of the people. If it once undertake these functions as its rightful province of action, it is impossible to say to it 'thus far shalt thou go, and no farther.' It will be impossible to confine it to the public interests of the commonwealth. It will be perpetually tampering with private interests, and sending forth seeds of corruption which will result in the demoralization of the society.
Its domestic action should be confined to the administration of justice, for the protection of the natural equal rights of the citizen and the preservation of social order. In all other respects, the voluntary principle, the principle of freedom, suggested to us by the analogy of the divine government of the Creator, and already recognized by us with perfect success in the great social interest of Religion, affords the true 'golden rule' which is alone abundantly competent to work out the best possible general result of order and happiness from that chaos of characters, ideas, motives, and interests—human society.
Afford but the single nucleus of a system of administration of justice between man and man, and, under the sure operation of this principle, the floating atoms will distribute and combine themselves, as we see in the beautiful natural process of crystallization, into a far more perfect and harmonious result than if the government, with its 'fostering hand,' undertake to disturb, under the plea of directing, the process. The natural laws which will establish themselves and find their own level are the best laws. The same hand was the Author of the moral, as of the physical world; and we feel clear and strong in the assurance that we cannot err in trusting, in the former, to the same fundamental principles of spontaneous action and self-regulation which produce the beautiful order of the latter
Whether the Democratic Review's policies, even in its early years, were consistent with this view, I leave to others. But it's pretty clear that Thoreau isn't to be credited with the phrase, unless one focuses on the precise placement of "best" in the phrase; indeed, he himself was referring to the phrase as an existing motto and suggesting a revision to it that would go considerably further.