Applied Philosophy: Who Do We Program This Car To Hit?
The ethics of autonomous vehicles.

In the last moments before your car crashes, you have just enough control over the wheel to make a choice: Are you going to collide with the motorcyclist who's wearing a helmet or the one who isn't? The guy wearing the helmet is more likely to survive the crash, so that seems like the less awful option, right?
Now suppose you're programming an autonomous vehicle and you need to decide how it should behave in such a scenario. Presumably it too should hit the guy whose head is protected. But wait: If that's the industry standard, doesn't that give bikers an incentive not to wear a helmet, increasing the likelihood that they'll be injured in some other, more statistically likely accident? You could try to keep the algorithm a secret, of course, but these things have a way of getting out—especially if someone sues your company after an injury. And there are ethical issues with not being transparent, too.
It's like a weird mash-up of the trolley problem with some Aaron Wildavsky–style study of regulation and risk, all wrapped up in one of Lawrence Lessig's arguments about code as law. But it's just one of several ethical dilemmas that the cars' designers face. Patrick Lin explores those questions in an interesting article for Wired. "While human drivers can only react instinctively in a sudden emergency," Lin writes, "a robot car is driven by software, constantly scanning its environment with unblinking sensors and able to perform many calculations before we're even aware of danger. They can make split-second choices to optimize crashes—that is, to minimize harm. But software needs to be programmed, and it is unclear how to do that for the hard cases."
It's inevitable that a lot of this is going to be determined by trial and error, as manufacturers make adjustments and a body of case law emerges. Obviously it's best to work out as much as you can in advance, but if that holds you up another issue intrudes: If autonomous cars are safer than cars controlled by a human driver, wouldn't it save lives to get them on the road sooner rather than later? So what are the ethical implications of an ethically motivated delay? Ow, my head hurts.
Editor's Note: As of February 29, 2024, commenting privileges on reason.com posts are limited to Reason Plus subscribers. Past commenters are grandfathered in for a temporary period. Subscribe here to preserve your ability to comment. Your Reason Plus subscription also gives you an ad-free version of reason.com, along with full access to the digital edition and archives of Reason magazine. We request that comments be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of reason.com or Reason Foundation. We reserve the right to delete any comment and ban commenters for any reason at any time. Comments may only be edited within 5 minutes of posting. Report abuses.
Please
to post comments
How will it determine the most racist, sexist, and anti-socialist person to hit?
Surely it will be able to read bumper stickers and apply stereotypes based on the make and model being driven.
Coexist!
For the longest time - I mean, like, till RECENTLY - I did not realize what that bumper sticker spelled out.
I was, like, "Canoeist...what??" Did. Not. Get. It.
Till I did - and then I snickered, both at me for being a doofus and not recognizing that for a couple decades, and for anyone that would put such a weak-ass sticker on an otherwise perfectly-good car.
They do it on the water.
If you paddle while nekked, are you a canudist?
Who the fuck are the Canutsons?
I much prefer the "coexist" stickers that use firearms manufacturer logos instead of various religious symbols. I suppose for some, though, that these *are* religious symbols.
*goes back to sipping whiskey and cleaning 1911s in the dark*
This is why I'm already headed toward gleaming, red Barchetta, and have a collection of vehicles with carbs and magnetos and shit. Fuck black boxes.
I appreciate the desire for collision avoidance SW, and embrace antilock brakes and (finally) airbags - but this auto shit is BOUND to fail eventually. So - Honda XR, Kawasaki ZRX and a coupla Jeeps in my stash with basically zero electronics on them. They'll still work when we're all Cuba in the US 70 years from now...
This is only an ethical quandary if you don't correctly apply the risk created by people deliberately not wearing helmets to game the software.
You try to game the software, that's your moral responsibility. If the things you do to game the software increase your own risk, fuck you, you lose.
Problem solved.
Someone not wearing a helmet creates a risk to themselves not me. I don't see how I could ethically consider a risk that I wasn't bearing as part of my calculus.
I should make the choice that results in the lowest chance of an accident. If I make it based on who is wearing a helmet, I either punish the guy wearing a helmet by hitting him to save the guy without one or I punish the guy not wearing one because I guess I don't like people who don't wear them. I don't see how either way of thinking is ethical. Each cyclist has the same moral value to me and I have an equal duty to avoid hitting them if I can.
I'd hit the fucker on the Harley, helmet or no.
If it's two Harleys, yeah, Ima try to take out both.
Of course the chances of two Harleys both running and having riders who are riding them versus just talking about them are about the same as winning the lottery and getting eaten by a bear on the same day.
It's true. One is usually driving around to get parts for the other.
I've heard biker gangs always keep an old Honda out back to run for parts.
If it's two Harleys, yeah, Ima try to take out both.
Way to find choice #3 in this game.
It's a non-issue to begin with.
While it's nice to thing that the on-board computers will record every sensor reading to the nano-second and flash it's memory to disk on every update, the truth is, it's impractical. Even if it were practical, sifting through the equations to figure out why it hit one biker rather than another would be like trying to figure out why Chris Dorner chose to hide in a cabin instead of sprinting to Mexico.
What you'll be left with is a collection of situations that trained the algorithm, suggestion of the conditions when the decision was made, and an engineer saying "Well, we programmed it to hit as few people as possible and then gave it 10,000 scenarios where it *had* to hit someone, and this algorithm had the fewest total kills."
Don't get a headache, Jesse. These cars will never drive the roads of America because, BANNED!!!!
Dilemma solved.
How can we have cars when we have no ROADZ because KOCHTOPUSTHUGLIHADRACISGENDEREDWARONWIMMINZ??!
This is why self driving cars are a terrible idea. Once they happen, it will just be a matter of time before they are mandatory. Once that happens, the government will just control how you drive on top of everything else. No thank you. I love driving and I will die someday of something.
Imagine, your stereo will automatically shut off at railroad crossings, even if a train hasn't come through your town in 40 years.
If the car is driving why would it care whether you can hear a train coming? Oh, right. Because Government.
Won't allow you to go over 55 mph. It would be horrible what these busybodies would do if they had the power.
well I'd then be late for everything.
And over night people would come up with a thousand different ways to hack these cars. You'd carry a little thumbstick with you to temporarily convert it back to the approved standard whenever you needed to do an inspection. Billions of dollars would be wasted trying to enforce the unenforceable.
Billions of dollars would be wasted trying to enforce the unenforceable.
Or as we call it, "Monday".
I disagree I think the state would hate this as it would cut into their highway tax, err I mean traffic safety ticket revenue.
You seem to think something other than your debit card will start the thing.
Now I have to find something to clean the coffee off my monitor... thanks. 🙂
I tend to agree, maybe. Self-driving vehicles will be a major moneysaver for things like warehouse transportation, mining vehicles, excavators, things like that. But on public roads, liability issues and government will ruin much of their promise.
I also think you're discounting the amount of people that hate driving and would be all for this, such as drunks and women. But your right if they go forward with this the only way it would work is if every car was self-drivable on the highway, otherwise liabilty would be to great. I envision a scenario in which all the interstate highway's are driverless only and the backroads/byways are drivers only.
I don't see a liability issue. Driverless cars are already much safer than the average driver. They don't follow too closely. They don't speed. They don't weave through traffic. They can see under and through other vehicles in 360 degrees. They pretty much never violate right-of-way and they are smart enough to slow down and stop whenever they can't handle their surroundings. They are actually better than most humans at interacting with humans on the roadway. I think you will start seeing big insurance discounts and tax breaks for people who use driverless cars. And the liability risk for manufacturers will be handled with insurance (or self-insurance).
Not to mention, when everything is done by computer, you'll be able to travel safely at double or triple the current pathetic speeds.
that could be, I guess we will see over the next 5 or so years when they start coming to market.
I might be for all driverless cars if it gets people to start turning from the correct fucking lane and get out of my way.
It is way too early to say that driverless cars are much safer than the average driver. Yes, the driverless cars that currently exist have an excellent safety record. But there are two things to note about this:
1. they punt control back to the human being whenever they get confused
2. they're being "driven" by people who are still treating them as testbeds and paying close attention.
I suspect things will look a little different once they start getting into the hands of people who just go to sleep behind the wheel.
I can't wait for automated cars (hopefully they will fly at several hundred knots as well).
The amount of time I waste driving is a crime.
They might even drive on your lawn.
Never mind that, wait until they start controlling where you drive!
Answer: go into a slide and take out both motorcycles.
No that is what you do if they are on bicycles.
Now suppose you're programming an autonomous vehicle and you need to decide how it should behave in such a scenario.
I say, choose the outcome most lethal to the passengers in the "autonomous vehicle".
"Are you going to collide with the motorcyclist who's wearing a helmet or the one who isn't? The guy wearing the helmet is more likely to survive the crash, so that seems like the less awful option, right?"
If you incorporate human agency into your ethics, then the guy who chose not to wear a helmet has willingly assumed more risk.
Based on my experience, the guy not wearing a helmet will be more aware of his surroundings, and better able to hear the fucking car coming, and at least attempt to react.
Based on my experience, the guy not wearing a helmet is deaf from years of windblast, and LOUDPIPESSAVELIVESDERP! and can't hear SHIT. Guy with a helmet on can still hear, and - having survived one off-bike excursion thanks to his/her helmet - takes evasive action JUST a tick sooner, saving self.
Your mileage clearly varies - that's cool.
I am with you. I used to go without a helmet sometimes when I lived in Texas. I found the experience generally unpleasant about about 30 miles and hour. The wind just beats you to death. The only time I would ever go without a helmet was when it was like 105 and I was heading a few blocks to dinner. Even that was stupid.
I hear the cars around me better when I'm wearing earplugs.
And any biker that depends on noise to tell him where the cars are is fighting hard for a Darwin award.
Rule#1: Assume you're invisible.
Rule#2: Assume all the drivers around you are drunk.
You've gotta see 'em. The noise around you is a lie.
I would change rule #2 to "Assume all drivers want to kill you at their first opportunity."
I spent a lot of time evaluating the distance between a car's tires and the lane markers.
And if you've ever spent any time in Rapid City in early August as a motorist, that's not far from the truth.
This. I rode daily in SoCal for 20 years, and never got tagged by a car. Moved to Idaho and hit a deer the second year. I used to bitch about how much money I had spent on crash gear until I went over the bars and hit the pavement. Icon kevlar leather literally saved my ass.
That's my assumption every time I ride. Many drivers are way too busy talking/texting/eating/shaving/make-uping to be bothered with actually driving.
I'd say you need to factor right-of-way in as well. You hit the guy who failed to yield right-of-way. You definately do not interfere with someone else's right-of-way just because it will create a "better" collision.
I'd hit the fucker on the Harley, helmet or no.
Bonus points for tattoos.
Wouldn't the most ethical decision be to do something like Batman did in Batman Returns? Have a mechanism that disintegrates the car before impact, but keep something left over to protect and transport the driver/passengers?
Now suppose you're programming an autonomous vehicle and you need to decide how it should behave in such a scenario. Presumably it too should hit the guy whose head is protected.
Who ever wrote the article is fucking idiot. Assuming that I am going to build a auto-driving system, we're going to be looking at vector math first. Mass and delta velocity (direction and speed). I don't give a fuck if the guy on the bike is wearing a helmet or not, all I care about is the total transfer of energy and momentum. The real choice is between hitting someone going in the same general direction I am (probably survivable) versus hitting someone coming straight at me (not survivable).
I assume, considering the forward momentum of the hypothetical car, you only have the choice of hitting something in your direction of motion.
No. Assume that I am making the choice between side swiping the guy next to me or running over the dick head pulling out of the parking lot.
All scenarios come with an implied ceteris paribus.
If you mean a totally concocted situation where the physics of hitting either vehicle is somehow identical, then I am running over the organ donor.
I would hit then both for hogging the road. Serves them right.
The fat guy. Always aim for the fat guy. Not only will he decelerate you more gently, but he's a fat guy, so fuck him.
Aren't all bikers fat?
The ones on Harleys. Much less common on sportbikes, but when you see a fat guy on a sportbike, it's hilarious.
I thought that was an age thing more than anything else. Trying to remember the last time I saw someone under thirty on a sportbike.
I'm 47 and I ride a Buell Firebolt. You wouldn't know my age because I'm covered head to toe and I'm not fat.
At 44, I'm one of the youngest guys in my Ducati club.
It is hilarious. The only thing more hilarious is a fat guy with his fat girlfriend on the back of a sport bike. I have only seen that twice in my life but it was comedy gold both times. You wouldn't think you could over tax a Ninja at highway speeds, but I think they proved you can.
Just head on down to SC for Memorial Day...
Have you ever seen a 350 lb person on a Honda motorized scooter, now that is hilarious.
Go fuck yourself.
"While human drivers can only react instinctively in a sudden emergency," Lin writes, "a robot car is driven by software, constantly scanning its environment with unblinking sensors and able to perform many calculations before we're even aware of danger."
I've found in my own riding that I tend to instinctively choose the option--when I have split second to choose--that I've thought about beforehand.
I had a farmer pull out to make a left turn in front of me on an empty highway in the foothills of the Sierras. I'd thought about that situation a lot beforehand.
If I try an emergency quick stop, my chances of a collision--just at lower speed--may increase, vis-a-vis a swerve, but my chances of surviving increase, as well.
If I swerve in front of the truck, the chances of surviving decrease--vis a vis a quick stop--but my chances of getting away without any collision at all increase.
There's no hard fast rule for which is better in every situation, but I've always thought that I'd rather swerve. What's the point of a more survivable collision--if that just means I end up in a wheelchair?
When that situation actually happened, in that fraction of a second, I swerved, and I think I more or less programmed myself to do that by thinking about it beforehand.
I can account for my own qualitative choices like software never can--and I'm self-programmable, too!
I try and stay in either the far right or far left lane as much as possible. That way if someone pulls in front of you, you can go into the shoulder or worst case into the ditch. Ditch is a really bad option but better than hitting someone.
I have been lucky, 30,000 miles of riding over a bit less than 10 years and I have never hit anything, laid the bike down or really had to make an emergency stop.
I keep telling people luck doesn't have anything to do with it.
I've been putting on 15,000 miles a year, and I've never been down either.
Some half of all motorcycle accidents happen when the motorcyclist is inebriated.
Since I don't do that, next take out all the people who are under 25, and motorcycling isn't anywhere near as dangerous as people think.
That's what I tell my girlfriends. Yeah, they're dangerous, but there nowhere near as dangerous as you are. I know chicks who can wreck a guy just by smilin' at him.
You're certainly safer on the back of my bike riding down a mountain highway than you are riding around Los Angeles in the passenger seat of a driver in his 20s.
I agree Ken. The biggest thing, beyond the obvious of people who don't know how to ride and wrap it around a telephone pole or people who drink and ride, is people don't pay attention to where they are in relation to other cars. The first thing I learned riding a motorcycle is that whenever someone cut me off it was nearly always my own damned fault for riding in their blind spot. It is not good enough to avoid running into someone. You have to stay where people can see you and out of people's blind spots.
The other issue is being careful on full access roads. People always think I am crazy because I will ride at speed on interstates. Interstates, if you have your head out of your ass, are actually quite safe. Most people who get in accidents get in them on side roads when someone doesn't see them and pulls out in front of them. Again, paying attention and being ready to go onto either shoulder if that happens is what keeps you alive.
Lastly, I am not a fan of cruisers. I think you sit too low and it keeps people from seeing you. My bike has a very upright position. Maybe I am kidding myself, but I think sitting higher and more upright is why no one has ever pulled in front of me. I am just easier to see.
+3
No. Chances are, if you're in a motorcycle accident it's a single vehicle accident. Yours. If the other is more statistically unlikely, why would you alter your behavior to account for it?
Helmets offer limited protection at low speeds, and not much at high speeds.
If nothing else, they can keep your face attached to your skull rather than being scraped off on the pavement.
I'm not so sure about that. If somebody rear ends you at a stop light because they're looking right through you (probably the second most common motorcycle accident), and you're going head first through the rear window of the car ahead of you, a helmet will make a big difference.
The other concern is about debris on the road while you're riding. Riding out through the desert, for instance, they're got these giant June bug lookin' things that hover at helmet height above the road. When you hit 'em and they get you in the chest, it hurts like hell--you take one of those to the head at 70 mph without a helmet?
Also, a helmet will protect your head from road rash if you're sliding down the freeway face first. I don't ride without one even when I'm in Utah--where you don't have to wear one.
Now I want to go back to Zion. That's it, I'm going to Zion this summer.
Writing as someone who would be dead otherwise, I'm going to have to say that you're mistaken.
And even if you were right, they'd still be better than no helmet at all.
How a Feynmanian physics doohickey that allows the car to calculate quantum mechanical probabilities both forward in time and backward in time in order to determine and avoid the very thing that caused you to crash in the first place.
What if the "cause" was a child that jumped out into the street? Would you allow your car to vaporize the child if it meant saving your life and the life of the hypothetical biker?
Objectively, children have no more value than adults.
Yes.
I mean, I'd prefer an outcome where nobody gets killed, and even the kid who just jumps in front of a ton or three of moving metal is unharmed.
But if it's a choice between "me and some innocent biker" vs. "kid who suicidally jumped in front of my moving - but not recklessly driven - car", I'm picking me and the other guy to live.
"But it's a kid!" doesn't override other people's lives.
Can't it be both?
From a purely theoretical standpoint? Maybe. Practically speaking? Not really.
I know many people who ride in my no helmet state because they want to. Their considerations seemingly go no further than that.
I don't think it would create any perverse incentives. The problem is the moral idea of me saying it is better to hit one person over another.
What's the alternative? Continue on course and hit whoever happens to be in that path?
Take whichever path creates the lowest risk of there being a collision. If you have to choose to hit one of the other, I don't think one having a helmet is a valid criteria.
What are valid criteria, then, given an either/or situation?
Well, if we're analyizing it to that level...we don't have enough information. I need to know the speed and location of all actors, and their relation to the road and surrounding obstacles.
The only valid criteria is the one that doesn't kill me.
Exactly or one that reduces the chance of an accident period. You don't judge which biker deserves to live or which one will be less injured if hit.
... so you don't aim to *minimize injury* in the case where a collision can no longer be avoidable?
Why?
So far I've only seen that asserted as if it's some sort of core principle, and it's one that baffles me.
I mean, by all means, *try to avoid a collision at all* as a primary principle.
But eventually there may be a case where it's unavoidable, and there are multiple outcomes to choose one.
The assertion that you simply must not take into account injury potential of the outcomes is baffling; I'd think that both as a consequential and intentional case that would be a primary consideration...
I think we'll see semi-automated cars long before we see fully self-driving cars. A self-driving car will have complicated liability, but something that recognizes an obstacle and stops more quickly than a human would be able to would not. It takes at least half a second to recognize a developing car crash ahead of you, for example. If you're going 68 mph, that's 100 fps, so cutting the reaction time by an order of magnitude would allow the car to begin braking almost 50 feet farther away from the accident. That's not a small thing.
We already have those types of semi-automated cars, don't we?
Do we? I know we have cars what will warn you if you're about to back into something, but I don't know if we have anything yet that will automatically break and swerve for you.
*brake* and swerve. Fuck.
The other day on Top Gear one of the guys was driving a car with radar controlled braking and cruse control. The cruise control thing was really cool. Here's a youtube of it. http://www.youtube.com/watch?v=eU83G3OaJuU
According to commercials we do.
So no.
Brake, yes. Swerve, no - the liability concerns are far too enormous just yet.
But there's no theoretical reason why such systems won't be coming, even without autonomous cars.
Google's self-driving cars have been operating on public roads in Las Vegas for three years now.
They've only been involved in one accident, last I heard, and that was when one of them was hit by a drunk driver.
There was one accident in SF in their testing. The driver took over for a while and rear ended a guy (or something like that - the point being that the human caused it, not the car). It was over 2 million miles in CA last I heard.
We are pretty close to that now. The new Mercedes Benzes have radars and will stop if the car thinks you are going to hit the car in front of you.
New Subarus are getting pretty close too. Rode in one the other day. Pretty much drives itself when in stop and go traffic.
http://www.subaru.com/vehicles.....tures.html
Hit the person who will minimize lawsuit damages.
This is obvious. Duh!
Ethicists really suck if they cant figure this out.
Hit the person who will minimize lawsuit damages.
In a truly connected world, the software in the car would identify you (say, from your license plate), and then be able to pull up who your attorney is.
No attorney? Hasta la vista, baby!
A retainer on file with Bloody, Knuckles, and Goons, PC? Trigger self-destruct sequence.
Which is preposterous and silly itself.
Ah, scarcity - how does that work, again? Surely, you're not suggesting that these ethically-minded cars are available to populated the roads in an instant.
Far more important than the Trolly Problem, how do we teach *anyone* (be they Human, half-Vulcan, or Computer) to beat the Kobayashi Maru?
I don't believe in the no-win scenario.
Aim for cars with federal or diplomatic plates first.
I'm thinking that "steering" at all is the wrong decision there, unless you can steer to avoid the collision entirely, because any portion of the traction circle you divert to maneuvers is traction that can't be used for braking. And since reducing the velocity of the collision is the thing most likely to reduce the harm overall, the best thing for the car to do is brake as hard as possible.
The only sane answer in the entire thread
You could make driving robot cars voluntary and have separate toll roads for them. I'm pro segregation of robot kind.
If that's the industry standard, doesn't that give bikers an incentive not to wear a helmet, increasing the likelihood that they'll be injured in some other, more statistically likely accident?
Naw, since most bike wrecks and rider head injuries aren't the result of a head-on with that level of flexibility involved.
(More relevantly, such a case as is ludicrously uncommon as to be irrelevant; it'd be one in several billion or trillion cases where there are both multiple motorcyclists and the car automation will be able to "pick one" without simply avoiding the wreck at all.
Now, the core question of "how do you make such choices to minimize damage to others and liability?" is unaffected - but I think we'll find most of the choices are much, much simpler; like perlhaqr said, mostly the solution is to stop the car as much as possible, not "pick your target".
With ABS and computer/camera systems detecting imminent impacts, it'll mostly be things you couldn't steer around in the first place that you'd be stuck hitting, I think.
And with extant systems already starting to detect potential stopped vehicles several vehicles ahead to allow for pre-braking, I suspect a far more likely outcome is simply a lot less crashing.)
"I'm afraid I can't do that, Michael..."
I will admit to LOVING fuel injection, and getting it on all my new vehicles.
However, when the Pockyclypse hits - ain't nonna that gonna work. So - Cuba. They've been driving those 50's 'murcan cars around with fucking carbs made out of Coke cans. Know how many people know how to do that? Fewer and fewer every year...
The Hot Rodders are the ones who become Lord Humungus' gang when the Pockyclypse hits...
OK, this is starting to get a little weird...
Flamethrowers - bullets keep going too far, might hit someone innocent
Whenever I read about EMP attack, it makes me happy to have an old American car. I could collect the gas from my other vehicles, load the wife, cat and retriever up into the Mustang and make for wherever the EMP attack didn't hit. Short of a full on North American apocalypse from a megaton blast over Kansas, I would probably only have to make it a few hundred miles.
EMP attack scares the shit out of me. Be nice if your governing class wasn't retarded and actually was doing something to harden the power grid in case of such a thing.
you called?
might hit someone innocent
Pfft - like THAT's a consideration.
*murderdrones a wedding, and then a kindergarten class*
Thanks. That poor orange Honda in the top picture.
It can happen from a solar flare. It will be darkly amusing if this is what causes the death of 90% of the worlds population.
During the Y2K scare, plenty of people would come into the local co-op talking about how they had generators, livestock (horses!!!!), and die-hard diesels. Almost invariably, they owned 1-2 guns and kept, at most, 100ish rounds.
Okay, so you've got plenty of food and supplies, generators to make power and let everyone within 20 mi. know you've got it all, and no way to defend it?
The worst aspect of an EMP attack is the sheer amount of free time other Americans will have to either figure out how to build an 'electrically hardened' vehicle or how to steal yours from you.
Our government could EMP proof the grid and eliminate t he risk for the cost of a single hopey changey stimulus package. That they haven't done that is so reckless as to almost be called murder if something ever does happen.
Stealing is a problem. That is why I have weapons.
I was reading something recently that said there was a flare capable of significantly fucking with the grid, just last year, and the earth just happened to be on the opposite side of the orbit.
Eventually it will happen.
It will. Hopefully not in my lifetime or if it does it hits the other side of the world and not here.
Thanks, I'd never seen it before.
http://underthehill.files.word.....oexist.png
Then there's this:
http://creepingsharia.files.wo.....ghtbox.jpg
Though it would still be appropriate if any symbol was slaying the others.