Facial Recognition Programs Are Getting Better at Recognizing Masked Faces

Time to add a hat and sunglasses!

Pandemic-wear face masks may not be quite as effective a barrier to surveillance as we hoped. While fabric face coverings certainly pose challenges to facial-recognition technology, U.S. government researchers say that improving software specifically intended to account for obscured features has reduced the error rate and made it easier to identify people whose faces are partially concealed.

So, you'll need to add sunglasses and a hat to your privacy-enhancing ensemble to confidently thwart the best efforts of the snoops.

"A new study of face recognition technology created after the onset of the COVID-19 pandemic shows that some software developers have made demonstrable progress at recognizing masked faces," the U.S. National Institute of Standards and Technology (NIST) announced this week.

The new report follows up on earlier research into pre-pandemic technology that found face masks to be effective at thwarting facial recognition efforts.

"How well do face recognition algorithms identify people wearing masks?" NIST asked back in July. "The answer, according to a preliminary study by the National Institute of Standards and Technology (NIST), is with great difficulty."

Since the earlier investigation, though, a new generation of facial recognition software has become available. The new technology is designed to be implemented in a world in which people's lower faces are routinely covered by fabric masks, obscuring many of the features that software would traditionally compare against stored images in order to identify subjects. The new systems "are able to recognize masked people by getting enough key points from just the eyes and nose," the South China Morning Post reported earlier this year.

They appear to do so pretty effectively, too.

"Some newer algorithms from developers performed significantly better than their predecessors. In some cases, error rates decreased by as much as a factor of 10 between their pre- and post-COVID algorithms," according to Mei Ngan, one of the authors of the latest NIST study. "In the best cases, software algorithms are making errors between 2.4 and 5% of the time on masked faces, comparable to where the technology was in 2017 on nonmasked photos."

As with the earlier study, NIST researchers applied digitally generated face masks over photographs in order to assess the new technology. They performed one-to-one matching, in which images are compared to stored photos of the same people—as is done at checkpoints or to unlock smartphones. A test of far more challenging one-to-many matching of the sort conducted when surveillance images of crowds are compared against large databases is planned for the future.

While researchers found that facial-recognition technology is improving in its ability to identify people through face masks, it still has limitations.

"When both the new image and the stored image are of masked faces, error rates run higher," they noted. "With a couple of notable exceptions, when the face was occluded in both photos, false match rates ran 10 to 100 times higher than if the original saved image showed an uncovered face." That's probably because fewer clear facial features yield fewer points of comparison.

They also discovered that, for uncertain reasons, the color of masks matters; "red and black masks tended to yield higher error rates than the other colors did."

Unsurprisingly, larger masks that obscure more facial features result in higher error rates than smaller masks that cover only the nose and mouth. "Continuing a trend from the July 2020 report, round mask shapes—which cover only the mouth and nose—generated fewer errors than wide ones that stretch across the cheeks, and those covering the nose generated more errors than those that did not." That makes sense for technology that is designed to accommodate masks by comparing details of the upper face.

"Prescription-type clear-lensed glasses will generally not be a problem for facial detection and recognition software. The key eye details are still visible," Cole Calistra, CTO of U.S. facial-recognition company Kairos, pointed out in 2015. "The best algorithms in the world still have difficulties, however, when people are wearing dark or shiny sunglasses, that effectively hides the pupils of the eyes. This problem is particularly compounded when the glasses obscure and hide the distance between the eyes, which is a key part of many algorithms."

Technology has advanced since then, but cameras still have to be able to see at least part of a face in order to match it against a stored image.

In China, where surveillance technology is eagerly deployed by the government, Hanwang Technology Ltd. moved early to tweak its software to accommodate faces masked against COVID-19. "But the system struggles to identify people with both a mask and sunglasses," Reuters reported in March of this year.

Throw in a hat with your sunglasses and mask and you have a combination that leaves even the best facial-recognition algorithms relatively little data with which to work.

Even more effective means are available for frustrating the snoops. But projectors that superimpose images of other people's faces over your own and makeup that distorts identifiable features are for the very privacy-minded and unlikely to see widespread adoption.

More practical are Reflectacles' glasses that enhance the already privacy-protecting qualities of shades with features that dazzle surveillance cameras. The latest versions are rather stylish and don't make wearers look like Elton John.

Despite advancing technology, face masks continue to have some anti-surveillance properties beyond their health applications. But for best results they need to be complemented by hats and sunglasses that give facial-recognition software little with which to work.