The Trillion Dollar Sure Thing, Or Business Consultants Say the Darnedest Things
Over at the Harvard Business Review, Washington University in St. Louis professor of strategy Anne Marie Knott has published, The Trillion Dollar R&D Fix, in which she claims that by short-sightedly under-investing in R&D American business executives are leaving trillions of dollars on the table. She comes to her conclusions by running a regression analysis aiming to uncover the effectiveness of R&D spending using data from 610 publicly traded companies from 1981 to 2006. Professor Knott finds:
A new metric for R&D productivity—which I call RQ, short for research quotient—can change all that. RQ allows you to estimate the effectiveness of your R&D investment relative to the competition and to see how changes in your R&D expenditure affect the bottom line and, most important, your company's market value. My research—which includes a comprehensive analysis of all publicly traded companies in the U.S.—suggests that if the top 20 firms traded on U.S. exchanges had optimized their 2010 R&D spending using the RQ method, the collective increase in market cap would have been an astonishing $1 trillion.
Amazing if true. However, over at The Atlantic Monthly blog, Jim Manzi, founder and Chairman of Applied Predictive Technologies, argues that Professor Knott's analysis is largely bunk. Manzi is the author of the insightful new book, Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics and Society (which I review in an upcoming issue of The American Conservative.) Manzi explains:
For example, Knott claims that she knows that Apple would have maximized its market value by spending $9.5 billion on R&D in 2010 They actually spent $1.8 billion. That's a fairly incredible claim. She thinks that what is generally conceded to be a management team that is pretty savvy about innovation underspent on R&D by more than 400 percent -- Apple ought to have quintupled its R&D spending in 2010. As another example, Knott claims that Dow Chemical could have roughly doubled its total market capitalization by increasing its R&D spending by 10 percent. That's a lot of money for them to leave on the table. And a very easy fix.
Knott claims that if just the top 20 American corporations had followed her recommendations, they would have collectively increased their market capitalization by more than $1 trillion. Consider this assertion for a moment The current total market capitalization of the top 20 U.S. public companies is a little over $4 trillion. Knott claims that she has outsmarted the entire system of management teams, investors, equity analysts, hedge funds, large-scale private equity firms and everyone else who is trying to change management practices to increase share price, and knows how to increase the total value of the most-closely followed companies in the world by almost 25 percent….by building a regression model using publicly-available data.
If you could rely on Knott's predictions, you could raise capital, buy these companies, change R&D spending in line with her model, and then sell them again at an enormous profit. You could start with Dow, because you know how to double its share price.
Maybe Knott has discovered an incredible, remediable market inefficiency, and somebody is about to get very, very rich. Or maybe there's a problem with her model.
Manzi shows that there is a problem with her model. It is no doubt the case that some companies do under-invest in R&D and that some over-invest, but can a one-size-fits-all formula really capture the information needed by a business executive to navigate his company through the competitive marketplace? In his new book, Manzi persuasively shows why everyone should be highly skeptical of results emanating from regression analyses of truly complicated social phenomena. The whole Atlantic Monthly Manzi post, There is no Easy Button for R&D, is well worth reading.
Editor's Note: As of February 29, 2024, commenting privileges on reason.com posts are limited to Reason Plus subscribers. Past commenters are grandfathered in for a temporary period. Subscribe here to preserve your ability to comment. Your Reason Plus subscription also gives you an ad-free version of reason.com, along with full access to the digital edition and archives of Reason magazine. We request that comments be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of reason.com or Reason Foundation. We reserve the right to delete any comment and ban commenters for any reason at any time. Comments may only be edited within 5 minutes of posting. Report abuses.
Please
to post comments
comet ot
is that anything like Halley's Comet
LH: comet typo is now history.
"Or maybe there's a problem with her model."
Well, the market seems to be firmly on this side of the question.
MARKET FAIL!!11!
If there's a problem with Knott's conclusion, it's because stupid reality refuses to align with her model.
I call bullshit.
""Manzi persuasively shows why everyone should be highly skeptical of results emanating from regression analyses of truly complicated social phenomena""
Exactly. You can build a 'model' to show absolutely anything. See: climate scientists...
I call bullshit on Manzi's math.
(9.5-1.8)/9.5 = 81%
Apple "underspent" by 81%, not 400%.
Denominator is the "correct" value, not the actual value.
Note: Im making no comment here on Knott's conclusions, which are probably bullshit too.
Ivory-tower moron says something moronic, film at 11.
-jcr
She's well on her way to becoming a Czar of something.
Knott claims that she has outsmarted the entire system of management teams, investors, equity analysts, hedge funds, large-scale private equity firms and everyone else who is trying to change management practices to increase share price, and knows how to increase the total value of the most-closely followed companies in the world by almost 25 percent...
Sounds like a politician in the making.
Since these people won't voluntarily take her unwanted advice, they should be forced.
Dear leader Anne Knott gives on the spot guidence to peoples chemical plant 'Dow'?
So basically Anne Marie Knott has never heard of the law of diminishing returns.
Beat me to it, Mr. S.
Beat me too.
Presumably they try to set their R and D budgets so that the marginal dollar spent returns exactly one dollar.
Using the return on all the spent dollars to predict return to spending beyond the marginal dollar traipses into la la land.
We also under-produced our annual quota of turnips and sturdy women's undergarments.
Holy shit, BOTH authors are idiots.
First, RD includes a shitload of trial and error. You can put it somewhere other than in the RD budget, but it's RD nonetheless.
Second, the payoff of RD is usually a lot longer the timeframe Knott claims. Whatever the RD budget of 2010, its value-add to the market cap won't be known until the end of 2014 or later. For fuck's sake, the market cap value of the iPad needs to go back to the RD of Jobs' "NEXT" tablet concept in the 80's.
and where the fuck are my ampersands???
The Federal Government swooped in and stole them in my post too. At least I assume it was them.
For other observers here, Voros invented the DIPS stat in baseball. In his original writeup, he put in a lot of "I don't want to overstate the value" and "I haven't done enough historical analysis" and "I'm not sure how much noise is in here", but that didn't stop a bunch of statheads from immediately claiming it was the best new work since the invention of the wheel and overusing the stat.
Um, not that I necessarily agree with Mrs Knott's assessment but there is precident for exactly this sort of thing happening.
Perhaps you should review the growth of Sabremetrics and the Statistical analysis of Baseball.
Essentially one guy who no one had ever heard of used statistical regression to prove that everything that highly successful General Managers and Scouts knew about Baseball was wrong and when a handful of teams (Oakland, Boston, New York)started using his theories they dominated for a decade.
Just because no one is using her theories does not prove she is wrong, it merely proves that there is an established orthodoxy which has not allowed her theories to have been tried yet.
You work for the government, don't you?
If we could just get The Right People, Top Men, with the Right Ideas and Proper Methodology, we could run business like....likes a SCIENCE! With PREDICTABLE results. It's just that everyone's out there doing THE SAME THINGS, and they're not looking at Ms. Knott's GENIUS ideas, which she's tried and proven and...ummm....
No - this is utterly different than "Moneyball".
For your next act, why don't you give us a dissertation on how "Freakonomics" is going to change things now that "The World is Flat"?
You will note that I specifically said that I did not agree with Knott's paper, I was explicitly commenting on the argument used against it, essentially that her paper is obviously wrong because if it were right then everyone would already be doing it.
While Knott is almost certainly wrong in her assessments this does not make the "why isn't everyone else doing it" argument an actual valid critique.
Further, using an invalid critique to counter an bad argument actually gives the impression that the argument is stronger than it is becasue your poor critique is easily refuted.
You want more examples?
Why were no governments liberal republics in 1776? Clearly liberal republics are an ineffective form of government because if they were then countries with hundreds of years of governing experience would have become liberal republics.
I can go on for hours with examples of transformative ideas that overturned existing orthodoxies in some way or another and make it look like I am proving Knott's argument in the process when all I am really doing is disproving the weak assed attack on it.
Rasilio|5.18.12 @ 3:07PM|#
"You will note that I specifically said that I did not agree with Knott's paper, I was explicitly commenting on the argument used against it, essentially that her paper is obviously wrong because if it were right then everyone would already be doing it."
Well, no. It's wrong since *no one* is using it.
There are some very bright people involved in this sort of stuff, and they are always and very interested in making money. Tons of it, if they can.
And you can pretty well bet that someone has tried the 'let's fund RD like there's no tomorrow' model, and you can see where it lead.
You can easily prove my comments wrong: As the article mentions, *if* she's right, someone can buy in (with their own money) and should get rich!
Please let us know how it works out.
Why would I do that when I have twice now said I believe she is wrong.
Now please explain why there were no liberal republics in 1776. Clearly if *no one* was using that governmental form and they had a lot of bright Ministers and Parlementarians and Philosophers working on the question of how to govern their fellow man and some of them had even tried it a few centuries earlier but no one was doing it thereby proving that a liberal democracy cannot work.
The point is not that Knott is right, the point is this line of reasoning is just as wrong as hers.
"Now please explain why there were no liberal republics in 1776."
Uh, are you having a 'dense' day?
Please explain the competitive market value of liberal republics.
The aguement made by Manzi is similar to the old economics "joke" - An econ professor and a student are walking across the quad when the student sees a $100 bill on the sidewalk. When he mentions this to the professor, the professor says that can't be a $100 bill, because, if it were, someone else would have already picked it up.
Just because nobody is using Knott's model doesn't, by itself, mean that the model is invalid.
It sounds foolish (the model), but until it's tried, or more rigorously tested, you can't just dissmiss it because no one is using it yet.
I think that's a valid arguement
No, Sevo, you're the one being dense. He said he thinks that Knott is wrong, but the counterargument is wrong too. See FlyoverCountry's joke for the explanation.
Sorry, but that's mostly bullshit.
For one, sabremetrics is vastly overrated as a predictive tool, some SABRE members hate the fact that the statistical analysis is used for more than historical analysis.
Secondly, the "growth" of sabremetrics is falsely attributed to recent work when it really goes back to Branch Rickey who basically turned minor-league baseball upside down by starting farm systems rather than buying player contracts from independent clubs.
Thirdly, the approach of Boston and New York was almost the exact opposite of Oakland. For all of Boston's talk about sabremetrics, their basic approach was Knott's - outspend other clubs on amateurs. Oakland's approach was to find value in amateurs that other clubs were overlooking. Whatever one can say about the Yankees' winning percentage, there are lesser-winning clubs with higher profit margins.
Sabremetrics may be overrated as a predicitve tool, however it is a damn sight more effective than some scouts "gut" feeling.
Further Sabremetrics is not about predicing which players will do well but rather what types of actions lead to wins and it's predictive value there is quite good.
That is, it can't say Ryan Braun will hit 40 Home Runs this year, but it can say that if by whatever method Milwaukee is able to collectively get on base 35% of the time then they will be an offensiver juggernaut whether or not Braun hits 40 Homers.
As far as New York and Boston's differences from Oakland, no they quickly adopted sabremetric analysis of players the difference is they had the money to buy the superstars or keep paying the ones they had once they got them wheras Oakland did not, however by the late 2004 all 3 teams were practicing some form of sabremetrics in their player evaluations and game strategies.
"...however by the late 2004 all 3 teams were practicing some form of sabremetrics in their player evaluations and game strategies."
Which immediately made sabremetrics worthless as a competitive advantage.
Yes, however it made avoiding it a key weakness. The first guy on board is helped greatly, the last guy on is hurt immensely, and in between is in between.
The big difference was that baseball was only making marginally poor decisions.
This is huge claims compared to that.
Heck, most of the stuff was known by some in baseball well before James started writing.
Earl Weaver was eschewing the bunt and playing for the 3 run homer in the 60s and 70s.
Innings pitched for starters has been on a decline for all of baseball history.
etc etc.
And pretty much everyone in baseball knew wins and loses didn't say much about how good a pitcher was. Nolan Ryan got the first million dollar contract in the late 70s despite consistently having a .500 winning percentage.
A player's contract should be based on how much revenue he generates more than how great his stats are.
Ryan is a perfect example of this. I think he was an above-average pitcher, but one cannot deny that the average baseball customer loved to watch him pitch and happily paid for the privilege.
He was much better than "above average". He pitches 5386 innings and had a career ERA of 3.19. Eight times in his career his ERA was under three while pitching over 280 innings.
What is a starting pitchers job if not to eat up innings while allowing very few runs? The Ryan was just pretty line of thinking is gold plated stupid.
The head of Core R+D at Dow Chemical said "Dow Chemical spends more money on chemistry research than all the universities in the US, combined."
There are only so many chemists around, just throwing dollars into the air will not magically give results.
Wait, thinking back to what he actually said, it was a bit more specific: "Dow chemical spends more money on chemical research than the Chemistry Department budgets of all US Universities, combined"
Research is a really difficult thing to do. It just kind of happens. You can't just put a bunch of people in a building and say "do research" and expect to get results. We need more "RD". Well sure, but what RD? Knott reminds me of an old Soviet Planner. If you just put in the right inputs the metric says we will get these outputs. Ah it doesn't quite work that way.*
*Ironically it works that way in one area, mathematics. The old Soviets loved mathematicians. The reason why is that you could lock a bunch of mathematicians in a building and they would in time actually come up with something new. This is why Russia still dominates the field to this day.
Apparently Dr Knott's undergrad degree was in mathematics.
If you could rely on Knott's predictions, you could raise capital, buy these companies, change RD spending in line with her model, and then sell them again at an enormous profit. You could start with Dow, because you know how to double its share price.
We used to have a way to get around the problem of uncertainty. We called it "religion". Anything that happened in the future? No need to worry about that. Divine providence permeated everything back then. Addressing uncertainty squarely? That was practically atheism to some people back then!
Nowadays, we're much smarter than that--people ignore the fact of uncertainty in much more sensible ways. We use college professors, public intellectuals, not to mention bureaucrats. There's no uncertainty anymore! We can replace it with regulation anytime we want.
The people on Wall Street and on our management teams; the people in our board rooms, they're not contending with uncertainty. That's just a ploy! They know exactly what's gonna happen in the future. Why, haven't you heard? JP Morgan took a $2 billion loss--just to spite the Obama Administration apparently. There needs to be a Congressional hearing!
Because we all know there's no such thing as uncertainty anymore. Does this professor really know how much R and D money should be spent? Of course she does! She's a professor. We should put in a new regulation requiring Fortune 500 companies to devote more of their revenue to R and D today!
Does this professor really know how much R and D money should be spent? Of course she does! She's a professor.
Then she should do what most of my professors did, consult. If her ideas have any merit, convince a company to use your model. If it works, the company and she would make a bundle. If not, then she (hopefully) learns something about the real world of business. Until she proves her model in the real world, it isn't worth the paper it's written on.
We should put in a new regulation requiring Fortune 500 companies to devote more of their revenue to R and D today!
I figure the way they'd go about it is to demand that the companies pay an "R 'n' D tax". The gubmint would supposedly use that money to do the RD or at least control it. They like control. Think of all of the opportunities for graft and corruption and punishment of political enemies.
We're missing out on $1 trillion in stimulus!
Here is the problem with Knott's hypothesis particularly in regards to Apple:
Most inventions make no money for the Inventor. Then somebody comes along, finds a better use for an invention, and makes a ton of money.
James Gardner calls it the Sidestep Twist. Jobs, Gates, and Wozniak did the first time when they ripped off Xerox's GUI and mouse, stuck then in personal computers, and sold them to the public. Xerox's RD, made them billionaires.
http://www.amazon.com/Sidestep.....445sr=8-1
Then Apple took their Mac operating system and shoved it into a smartphone (thanks Palm and RIM) - and they made billions.
There are many other examples - companies like Garmin took the clunky GPS used by the military, supperimposed roadmaps, and created a multi-billion dollar industry. Thanks for the sattelites and RD Uncle Sam!
The money is in the practical application of technology not the development of the technology itself. The makers of electronic household appliances made a whole lot more money than Edison or Westinghouse ever did.
"She comes to her conclusions by running a regression analysis aiming to uncover the effectiveness of RD spending using data from 610 publicly traded companies from 1981 to 2006."
Yikes. We're actually in the ballpark of the area of my expertise (or at least the closest thing I have to an expertise), and while I'm sure there's more to it than that, she's got to know that folks will bring up the "correlation does not imply causation." Showing a correlation between RD spending and future revenues doesn't really do anything to help you figure out how much RD spending causes those increases. The "third variable" problem would be huge here, I would think.
And the causation could go the other way. In fact it probably does. Companies that make more money spend more on RD because they have it. RD is the first thing to cut when hard times hit.
Well I'm guessing she probably did at least try and control for that in the regression (regressions are nifty little tools to be sure), but where it becomes more difficult is in trying to isolate all the possible variables in the world that could affect both a companies willingness to spend on RD and their likely future revenues.
In any event, independent real world projecting would be necessary regardless of how she built her model before she could make any conclusive claims. IOW, if the model works as well as she says, she should be able to predict future revenues from current data with a relatively decent amount of accuracy, or at least better accuracy than is done without it.
That is a great point. And if you can predict future revenues, you can pretty much own the stock market and buy yourself a custom 747 and trip to the moon on Virgin spaceways out of petty cash.
Since I seriously doubt she will be doing that, her regression is probably bunk.
That is a great point. And if you can predict future revenues, you can pretty much own the stock market and buy yourself a custom 747 and trip to the moon on Virgin spaceways out of petty cash.
Since I seriously doubt she will be doing that, her regression is probably bunk.
John Voros McCracken: Your observations are astute and they are the same ones made by Manzi in his rebuttal.
Damn! My ampersands are disappearing too.
Damn you Eric Holder!!
Well honestly, I clicked on the Manzi article, but then had a tl;dr moment.
She probably just wants to meet her hero Paul Krugman.
Yep, those hybrid drive systems and fuel cells just invented themselves. Surely we'd have done that decades earlier if we'd have just thrown MORE money at it FASTER.
Cause the contraint was MONEY, not all the other factors associated with R 'n' D.
What a simple fuck. She'll make an excellent bureaucrat. First, I'd like to see her go supervise the afternoon shift at an auto or parts plant, and then do about another 6 months in product development, and THEN come tell me about this subject. But she's so smart, she can just bypass those steps...
Don't you understand that we could all have our jetpacks by now if the government had just thrown enough money at the problem?
I have to wonder what the current fucked-up state of patent law and administration is doing to R D budgets.
I would think it would impose a real threat to any return on your R D. You're thinking about spending a billion to develop something, only to find out some patent farmer is will sue your ass if you bring it to market.
One can spend a shit-ton on shiny, new RD, or one can spend a lot less for slightly used RD from a motivated seller.
"She comes to her conclusions by running a regression analysis aiming to uncover the effectiveness of RD spending using data from 610 publicly traded companies from 1981 to 2006."
I wonder if it might have occurred to Knott that there might be just a little bit of a causality problem in her regression. A mature industry is going to have little opportunity for growth. Any manager who isn't an idiot will cut back RD spending. A company in an expanding industry will likely spend heavily on RD just to keep up with the evolving market and encroaching competition. All of a sudden, Knott's regression follows.
It does stick out there like a sore thumb, doesn't it. She likely will claim she's controlled for all the possible third variables out there, but in my experience you never really can.
The model doesn't work until it works out in the wild, and even then it might work for a while and then cease working once everyone starts "teaching to the test" so to speak.
This kind of predictive statistics is quite tricky and will make you look foolish, but I find the field (predictive statistics) just fascinating.
As for the subject of sabermetrics, "not with a ten foot pole."
Sabremetrics is just geeks inventing their secret lingo for baseball. See, you non nerds don't know what you are seeing. You think driving in a run or striking someone out is a big deal. We nerds know different.
That is all it is.
It's the acronyms that really bother me...
She thinks that what is generally conceded to be a management team that is pretty savvy about innovation underspent on RD by more than 400 percent -- Apple ought to have quintupled its RD spending in 2010.
I don't know much of anything about statistics so I cannot comment intelligently on the details of Knott's analysis. But I'm wondering: what research should Apple have spent all of that money on? It seems to me that an analysis that can't answer that question is of limited value at best.
It makes me think that Knott is a disciple of the Krugman Temple: this sounds suspiciously like a claim that paying people to dig useless holes and fill them up will stimulate the economy.
After reading all the one-sided comments, I feel the need to step in here and defend Knott and her simplistic model and predictions. First of all, as for all the criticisms of her model I would ask you to describe to me your superior model. Further, what does your model predict? Just spend the same rate as the everyone does now? That is indeed what our herdlike business leaders do. Then they acquire startups left and right to make up for their shortsightedness.
As for arguments about market efficiency, the irony of the situation is that the more successful a firm is, the more they will want to resist progress. Because as long as that happens they have a risk-free lead. Everything else is risky. And as an aside, speaking of "free" markets, I wonder how much more the top firms would spend on research funding if they didn't have options like the patent court to "outcompete" upstarts. I bet it'd be quite a lot more.
Anyway all you need is a different tolerance for risk and value for short-term versus long-term profits, to make completely different decisions for investment. Doesn't prove anyone is "wrong".
Thanks