Government Offers Weak Defense of Bogus Study that Fueled Chinatown Bus Closings

A 30-year-old man wearing only his underwear was lying in a fetal position in the middle of a highway in Weirwood, Virginia, when a passenger bus struck and killed him. It was 1:30 in the morning. The bus belonged to a company called Virginia Seagull Travel, which is a "curbside" bus company, meaning it picks up and drops off passengers on a street curb instead of at a traditional station.

This tragic accident, which took place on July 9, 2008, had outsized influence on the findings of an error-riddled government study comparing the safety records of curbside vs. conventional buses, which I wrote about for Reason.com a couple weeks ago. The National Transportation Safety Board, a federal agency, published the report.

Let's even grant that this particular accident could conceivably tell us something meaningful about the safety of curbside buses, even though an official investigation absolved Virginal Seagull Travel of all responsibility. With regard to this accident, the NTSB study's main sin from a statistical standpoint was the fact that it ignored that the company is a tiny outfit. Over the course of six years, Virginia Seagull Travel maintained a fleet size of just three buses and had this one fatal accident. The NTSB converted these numbers into an accident rate of 35 fatal accidents for every 100 buses. A much larger company running 1,515 buses and 21 crashes in which someone was killed was assigned a rate of 1.3 fatal accidents for every 100 buses. Ludicrously, the large company was weighted the same as Virginia Seagull Travel. By disregarding wide variations in company size, the NTSB report arrived at a headline-grabbing and misleading conclusion that curbside bus companies were "seven times" more fatal-accident prone than conventional bus companies. By analogy, as I wrote in my article, "it’s as if a rookie baseball player with three at bats and one hit received the same ranking as a starter with 600 at-bats and 200 hits."

This instance of "statistical malpractice," which is how quantitative analyst and statistics expert Aaron Brown characterized it, isn't even the report's most egregious flaw. It also misclassified bus companies. Inexplicably, Peter Pan and Greyhound (among several other conventional carriers) were counted as curbside companies. Also, the NTSB press office trumpeted statistically insignificant findings, including the "seven times" figure mentioned above.

Now other reporters have taken a second look at the report and the National Transportation Safety Board has responded to several points in my original article.

Bloomberg News reporter Jeff Plungis, who covers the bus industry, has a piece out headlined, "NTSB Defends Study that Preceded Chinatown Bus Safety Sweep." Plungis writes:

The U.S. National Transportation Safety Board is defending itself against accusations it skewed a study that preceded the shutdown of 26 so-called Chinatown bus operations in the Northeast.

Anomalies in how the board classified motorcoach operators and calculated fatality rates raise doubts about its conclusion that curbside companies, including most Chinatown carriers, are about seven times more probable to have passenger deaths than companies using terminals, according to the Reason Foundation, a Los Angeles-based advocacy group for limited government.

The article goes on to state incorrectly that one of my main criticisms was that the study didn't factor in mileage when calculating accident rates. Mileage counts aren't in the raw data, so there would be no way to factor them in. Then it leaves out mention of how the NTSB distorted its findings by blending accident rates of large carriers and small ones without weighting by size.

This omission allows the NTSB to get away with making hay out of the fact that reclassifying the bus companies doesn't, as the article notes, "alter the finding that curbside operators have been more dangerous." Technically it's true that if you correct the data by moving nearly all the fatal accidents from one column to the other it doesn't dramatically change the results. That's because of the very methodological flaw the article didn't mention: The companies weren't weighted by size, so many of the accidents have a miniscule effect on the results. Aaron Brown put it best: "If virtually all your data doesn't affect your conclusion, you're doing something wrong."

Over at Next City, Matt Bevilacqua offers a pretty good summary of the NTSB report's shortcomings, noting,

In a long article published last week, Reason magazine’s Jim Epstein draws attention to what he considers a number of major flaws in the study. The most glaring of these is the fact that of 37 fatal crashes cited by the NTSB, a full 30 didn’t involve curbside buses at all. Rather, “conventional” bus companies, or those with stops at actual stations, accounted for most of the accidents. The well-known conventional bus service Greyhound, for instance, was responsible for 24 of the crashes.

Bevilacqua does allow the agency's press officer, Eric Weiss, to get away with claiming that I put too much emphasis on one chart: “'[Epstein] pulls one of nine charts on one page of a 66-page report,” Weiss said. “That’s not even one of the goals of the report, and he says that it is the centerpiece of our report.'" He's referring to the appallingly inaccurate finding that curbside bus companies are "seven times" more fatal-accident prone.

Surely Weiss is being disingenuous, since he knows full well that his organization chose to highlight this chart in its press release, which in turn inspired major newspapers around the country to repeat this absurdity either in their headlines or at the top of their stories (e.g., "Chinatown Buses' Death Rate Said Seven Times That of Others").

Anyway, as I noted in my piece, while that particular chart is particularly offensive the rest of the report is just as meaningless. If your data are wrong, your analysis is wrong across the board.

Bevilacqua also got Weiss to utter this howler in response to my charge that the NTSB reported statistically insignificant findings: "'We do not say it is statistically significant, we just put out the numbers.'”

To demonstrate the absurdity of that statement, let's assume the NTSB had looked only at Virginia Seagull Travel on the day it happened to have a fatal accident. Then it would have found that curbside buses have a 100% fatal accident rate. If that were the case, would Weiss also have just "put out the numbers?" Just "putting out numbers" without mentioning that they're practically meaningless is profoundly irresponsible.

A piece in The Weekly Standard recounts how the National Transportation Safety Board denied my request for the study data:

As [Epstein] was researching his piece, he asked the NTSB to share the data. They did not respond. So he filed a formal FOIA request for it. That was ignored, too. In particular, Epstein was interested in a chart showing the relative accident rates and confidence intervals. The NTSB told him that such a chart did not exist.

So my employer had to purchase bus accident data from a federal contractor and I had to rebuild the chart myself. Shortly after my article came out, the NTSB released the "nonexistent" chart.[*] Eric Weiss apologized to me via email, citing a "misunderstanding." As The Weekly Standard notes, "If you find that infuriating, imagine how it must feel to have been involved in one of the 27 curbside companies that was shut down" after the flawed report was published.

[*] The chart is hard to find on the NTSB's website, so I've reposted it here. It fully substantiates my case against the study. The NTSB's numbers do differ slightly from mine because the federal dataset they're drawn from is updated retroactively on a daily basis. I accessed the data about a year after the NTSB, so my numbers are more up to date.

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of Reason.com or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  • Scruffy Nerfherder||

    So who is the NTSB working for I wonder?

  • Almanian!||

    Dead guys who once were lying in the middle of the road, apparently.

  • ||

    It's called "regulatory capture".

    How much do you want to bet the people at the NTSB have connections to the larger bus carriers? They may have been employed as lobbyists for them or actually had worked for them in the past.

  • Libertymike||

    That is one reason why one should always dismiss the "findings" of a study sponsored, even in part, by any person or entity who

    (a) works for the state
    (b) depends upon the state
    (c) is a crony capitalist
    (d) wants to impose their way upon others via the state

    Any argument to the contrary is pure poppycock unsupported by reality and human nature.

  • Jordan||

    See "public health" research.

  • sarcasmic||

    Never let the facts get in the way of the narrative.

  • deified||

    So the federal bureaucracy NTSB basically homicided 27 private companies who had done nothing legally or ethically wrong, thus simultaneously obliterating jobs and increasing prices for the consumer?

    Good job, Federal Government.

  • Almanian!||

    Fucking up stuff - it's one of the things the government is really good at.

  • tarran||

    They also come into our homes and wreck up the place too.

  • Almanian!||

    Nice little dog ya got there...

  • ||

    Right. In a just world, the owners of these businesses would be able to sue for redress and would be handsomely compensated for what was done to them.

  • JW||

    I an just world, those mendacious hacks at the NTSB would be swinging from lampposts.

  • ||

    "If you find that infuriating, imagine how it must feel to have been involved in one of the 27 curbside companies that was shut down" after the flawed report was published.

    Equally infuriating, I'm sure there will be idiot progressive out there who will just think "CORPORATIONS!!!!" and assume that the people who started those companies are just rich capitalists who don't deserve a fair shake in the market.

    Most likely the people starting these companies are immigrants and small businessmen, who invested everything they had in it and are now financially ruined. Many of them will lose their homes. They will be forced into bankruptcy. Some will end up on public assistance, although, given that they've proven an ability to work hard and build a business from scratch most will probably recover. Unlike the shit-bag progressive morons who will read this article and just decide it's an evil Koch conspiracy by virtue of the fact that it was published by a libertarian.

  • Loki||

    Some will end up on public assistance

    Feature, not bug. /prog-derptard

  • Hyperion||

    NR, my typical modus operandi as of late. Not sure if this has been discussed here as work is not allowing me much posting time. But the Swedes have really jumped the shark. The city is being slowly burned down, and their level of PC is just off the charts. I'm not Islamaphobic or anything like that, but I'm not stupid either.

    From a recent BBC article:


    "My colleagues say the people on the streets are a mixture of every kind of people you can think of," he added.

    "We have got Swedes, we have got very young people, we have got people aged 30 to 35. You can't define them as a group.

    "We don't know why they are doing this. There is no answer to it."

    Stangely followed by the next paragraph


    In Husby, more than 80% of the 12,000 or so inhabitants are from an immigrant background, and most are from Turkey, the Middle East and Somalia.

    They know who's doing it! What is wrong with these people?

  • Scruffy Nerfherder||

    What is wrong with these people?

    Moral relativism combined with the European version of white guilt.

  • Jordan||

    Dude, it's the country where men piss sitting down and girls aren't allowed to play with dolls and boys aren't allowed to play with toy guns because of the dreaded GENDER ROLES. Let it burn.

  • KPres||

    What's wrong with these people? Who gives a shit? It's a religious war with fundies on both sides acting like animals. And there's nothing more delicious than watching religious fundies destroy themselves with their own stupidity.

  • ||

    You could maybe follow this up with a human-interest story about what happened to the people who used to own and operate these companies. How much money they lost. How their kids won't be able to go to college etc. People need to know the human cost of this kind of corrupt regulation.

  • JW||

    The diner never interviews the eggs that make up his omelette.

  • Libertymike||

    "Studies show".................

    WEAK.

  • Hyperion||

    Same old story, different victims of cronyism. This is just the same story as the Stossel piece about people trying to rent rooms, and the big hotels crying to their cronies to get them shut down. Same thing here. It's just cronyism all the way down. We should just rename the USA to UCSA.

  • johnl||

    Can I ask someone to put that data into plain text? Plain text, not Excel, is the lingua franca. That workbook just isn't opening for me.

  • johnl||

    OK I got it but people will have problems with Excel instead of text.

  • Raston Bot||

    Does anyone else want to party in Weirwood?

  • JW||

    Since the NTSB report came out, the federal government has shut down 27 Chinatown bus companies.

    Those dangerous, rogue privateers were taking the food right out of the mouths of the families of hard working Amtrak employees.

  • the other other alan||

    There is only one appropriate response to this article: "Yes, Mr. Bevilacqua."

  • Krogoth||

    What kind of company goes by "Virginal Seagull Travel" ? I am not sure I am comfortable with any of the implications.

  • owen||

    Hello, friends! Welcome to http://modernjerseys.org/. We offer cheap nice nike,addidas,reebok jerseys, caps and so on. We have updated a lot of jerseys http://mallsjersey.blogspot.com/ such as: NFL, MLB, NBA, women jerseys and youth jerseys. If you pay by Western Union , you’ll get 10% discount. We can give you the best discount. If you have any question please ask me or email me. Thanks a lot for looking around and ordering.

GET REASON MAGAZINE

Get Reason's print or digital edition before it’s posted online

  • Video Game Nation: How gaming is making America freer – and more fun.
  • Matt Welch: How the left turned against free speech.
  • Nothing Left to Cut? Congress can’t live within their means.
  • And much more.

SUBSCRIBE

advertisement