by Michael Grabell and Christian Salewski, ProPublica
While X-ray body scanners used in airports face concerns about potentially increasing cancer cases, a safer type of scanner has been plagued by another problem: a high rate of false alarms.
The scanner, known as the millimeter-wave machine, uses low-level electromagnetic waves that, unlike X-rays, have not been linked to cancer. The Transportation Security Administration already uses the millimeter-wave machine and says both types of scanners are highly effective at detecting explosives hidden under clothing.
But two of Europe’s largest countries, France and Germany, have decided to forgo the millimeter-wave scanners because of false alarms triggered by folds in clothing, buttons and even sweat.
In Germany, the false positive rate was 54 percent, meaning that every other person who went through the scanner had to undergo at least a limited pat-down that found nothing. Jan Korte, a German parliament member who focuses on homeland security, called the millimeter-wave scanner “a defective product.”
While it’s difficult to know for sure if the millimeter-wave machine has a worse false-alarm rate than the X-ray machine, recent tests suggests that it does. The TSA wouldn’t release its results, citing national security. But a British study found the X-ray machine had a false-alarm rate of just 5 percent.
For the millimeter-wave machines, a complicating factor is new privacy software that was installed in many countries after a public outcry over the scanners’ graphic images. The software automates detection and no longer creates an image of a passenger’s body. While false alarms were reported before automation when human screeners interpreted images, the software appears to have made the problem worse.
The privacy safeguards are also an obstacle to lowering the false-alarm rate, researchers say. The machines do not save images or data, which could be used to teach the software how to distinguish real threats from false ones.
The problem of false alarms comes down to fundamental physics. Millimeter waves penetrate clothing and reflect off objects. But because of their frequency, millimeter waves also reflect off water, which can cause the scanner to mistake sweat for a potentially dangerous object, said Doug McMakin, the lead researcher who developed the millimeter-wave scanner at the Pacific Northwest National Laboratory. (X-rays, which operate at a higher frequency, pass through water more easily.)
In addition, millimeter waves penetrate clothing materials differently, and layers of clothing can create a barrier, triggering a false alarm.
“These are known as clutter issues in the imaging,” McMakin said.
The manufacturer, L-3 Communications, said that in the United States the scanners have not experienced a high rate of false alarms caused by either clothing or sweat. L-3 executives noted that the millimeter-wave machine is installed in airports in some of America’s most humid cities, including Houston, New Orleans and Miami.
But as late as last November, the head of the TSA told Congress that false alarms were too frequent to deploy the privacy software. The TSA said the rate has improved since then and now meets its standards, which it would not disclose.
“As with many types of technology, there will be an anticipated amount of false alarms that are considered acceptable, and we continue to work with industry vendors to improve both the detection and operational capabilities for all of our technology,” spokesman Greg Soule said.
But results from other countries, as well as tests conducted in the United States before 9/11, show false alarms occurred between about a quarter and half of the time. Moreover, dozens of U.S. travelers told ProPublica they had to get a pat-down despite passing through the body scanners.
Only one report of the false alarm rate for the X-ray body scanners could be found. At Manchester Airport in the United Kingdom, where 13 machines have been tested on more than 2.5 million people, the rate has been less than 5 percent — and that includes passengers who left items such as keys in their pockets, said airport spokesman John Greenway.
Referring to the false alarm rate, Peter Kant of the manufacturer, Rapiscan Systems, said, “Our numbers internally are in the very low single digits.” The company, as well as several physicists, said sweat does not cause false alarms with the X-ray scanners.
In an effort to close a gaping hole in its ability to catch explosives, the TSA in 2009 began installing body scanners alongside metal detectors for routine screening. The deployment ramped up quickly after a Nigerian man tried to blow up a plane that Christmas with explosives hidden in his underwear.
The TSA purchased both types of scanners with plans to deploy them at nearly every security lane by 2014. In hubs, such as Atlanta and Dallas-Fort Worth, it installed millimeter-wave machines, which look like round glass booths and emit low-powered electromagnetic waves similar to those found in police radar guns.
In other major airports, such as Los Angeles and Chicago O’Hare, it installed X-ray machines, also known as backscatters, which look like two large blue boxes and emit extremely low levels of ionizing radiation, a form of energy that strips electrons from atoms and damages DNA, potentially leading to cancer.
The possible health risk of the X-ray scanners, while small, has prompted several prominent radiation experts to ask why the TSA doesn’t just use the millimeter-wave machine. The agency has said keeping both technologies in play encourages the contractors to improve their detection capabilities and lowers the cost for taxpayers.
The United States is almost alone in deploying the X-ray body scanners for airport security: Nigeria has installed them, and the United Kingdom is testing them for random screening and to check passengers who have set off the metal detector. Last month, the European Union prohibited the X-ray machines, effectively leaving the millimeter-wave scanner as the only option in Europe.
The United Kingdom will have to stop using the machines once its test is completed, according to the European Commission. But the commission has also asked one of its scientific committees for a health study that could change its position on the backscatters.
Guns, Sweat and Privacy Fears
During a Republican presidential debate in 1988, George H.W. Bush, pulled out a .22-caliber miniature revolver made with only a small amount of metal to dramatize the new types of guns that could pass through airport metal detectors.
“That weapon at this point cannot be detected,” he said. “That weapon can kill the pilot of an airplane.”
The comments, along with concerns over a new Glock pistol made of plastic, spurred the Federal Aviation Administration, which was then in charge of security, to fund research into a millimeter-wave imaging system at the Pacific Northwest National Laboratory.
After 9/11, the lab licensed the technology to a startup company, which was acquired by L-3 in 2006.
When the scanners debuted, TSA officials boasted that they were so good at detection, that screeners could literally see the sweat on someone’s back.
At that time, human operators viewed the image. Although sweat might appear similar to a threat, trained officers learned to recognize normal sweat patterns, said Kip Hawley, TSA administrator from 2005 to 2009. In fact, sweat could help officers detect a sheet explosive, he said, because something taped or glued to the body changes the natural sweat pattern.
“It never popped up where we said, ‘Oh God, we’re getting killed with false positives,'” Hawley said. “I think it’s a training issue, training the officers on interpreting the images.”
But because of the uproar over agents seeing passengers’ bodies — what critics decried as a “virtual strip search” — other countries began installing automated detection software last year, and the TSA followed suit in July.
Now, instead of displaying an image of a particular passenger’s body, the machine shows a generic, unisex outline that’s reminiscent of the cartoon character Gumby. Any potential threat is indicated by a yellow box that shows up roughly where the software detected it — on the right ankle, for example, or the left elbow.
“It looks for abnormalities,” said Tom Ripp, president of L-3’s security and detection division. “It looks for objects that are not supposed to be there.”
The advantage, L-3 officials said, is that screeners can focus their checks on the highlighted area instead of patting down a passenger’s entire body.
“If you go out to an airport like D.C., Reagan, you’ll see how easily the process works,” said senior vice president Bill Frain. “Usually somebody left something in their pocket. We sat there and watched for 20 minutes. The duration between an alarm and a check — they were just putting people through. It was a very quick check.”
The European Experience
That wasn’t the case in Germany.
The German interior ministry tested two L-3 body scanners with the automated detection software at Hamburg Airport, screening 809,000 airline passengers from September 2010 through July 2011. Despite the high rate of detection, the delays caused by frequent false alarms were so unbearable that Germany decided that the technology was not ready for everyday use.
Nearly seven out of 10 passengers had to be stopped for further screening. Although some passengers had forgotten coins or tissues in their pockets, 54 percent of all passengers who went through the scanners triggered true false alarms — meaning that no hidden objects were found on those people, a ministry spokesperson said.
The vast majority of false alarms, affecting 39 percent of all passengers, were attributed to sweat, buttons or folds in clothing. Another 10 percent resulted from passengers moving during the scan, while 5 percent couldn’t be explained at all.
Ripp from L-3 said the high alarm rate comes down to how diligent the screeners are about asking passengers to take off belts and boots, remove bulky sweaters and assume the proper stance with their hands over their heads. In the United States, the stance has become routine, he said.
“That was not the case in these trials in Hamburg,” Ripp said.
The German interior ministry, however, dismissed the idea that it hadn’t followed the manufacturer’s protocol. Officials there provided ProPublica with a flier [7] that was handed out to passengers before the screening that specifically tells them how to stand and to remove sweaters, belts and boots.
“Prior to the field test, the security personnel was specially trained to deal with body scanners and has adhered to the control procedure,” the spokesperson said via email. “The passengers were asked to take off the named items.”
Germany wasn’t the only country to have problems with false alarms.
France tested the scanners with and without the privacy software on more than 8,000 passengers flying out of Paris’s Charles de Gaulle Airport to New York from February to May 2010. But the government decided not to deploy them because there were too many false alarms, said Eric Heraud, a spokesman for the French civil aviation authority.
Heraud wouldn’t release specific figures but said the false alarm rate was higher with the automated detection than when officers interpreted the images. France plans to conduct a new test of the millimeter-wave scanners in 2012.
In Italy, the rate of false alarms was 23 percent, said Giuseppe Daniele Carrabba, head of the airports coordination department for the Italian civil aviation authority.
Italy tested two L-3 scanners with the automated detection software at the airports in Rome and Milan. The test ended in September, and officials are awaiting a final decision on whether to deploy the machines later this month. Carrabba said he thinks Italy will use them, and that the false positive rate will improve with more training and better preparation of the passengers for screening.
L-3 attributed the variations in experiences to the different settings that countries choose for what to detect and what to ignore.
Other countries that have deployed millimeter-wave scanners — Canada with 51 machines and the Netherlands with 60 — said they had not experienced problems with false alarms. They declined to disclose their false-positive rates.
The American Experience
In the United States, the TSA has deployed more than 250 millimeter-wave machines and plans to install 300 more by next spring.
The TSA declined to answer detailed questions. Instead, the agency released a statement saying that it had tested the automated detection software rigorously.
“Once it met the same high standards as the technology currently in use, TSA successfully tested the software in airports to determine whether it was a viable option for deployment,” the statement said. “While there are no silver bullet technologies, advanced imaging technology with this new software is effective at detecting both metallic and non-metallic threats.”
Shortly after the machines were developed, preliminary tests at Seattle-Tacoma International Airport in 1996 resulted in a false alarm rate of 31 percent, according to a research paper presented at a conference the following year. During the tests, screeners who were new to the machine viewed images of people carrying various weapons, explosives and innocuous objects and had up to 27 seconds to identify them. According to the paper, researchers did test the results with layered clothing.
In 2000, those same images were run through a primitive model of the automated detection and privacy software. The false alarm rate increased to 38.5 percent when the machine was set on high sensitivity but decreased to 17 percent when set on low sensitivity, according to another study by the same researchers at the Pacific Northwest National Laboratory.
“Overall, these results show comparable performance” between the software and the human screeners, the researchers concluded.
The TSA ran additional tests over several years before deploying the scanners, but late last year, administrator John Pistole told Congress the tests were still showing a high rate of false alarms with the software. Officials said the false alarm rate improved, and the agency began installing the software over the summer.
Still, American travelers frequently complain about false positives similar to those experienced in Europe.
Lynne Goldstein, an archaeologist at Michigan State University, said she generally prefers the scanners because, with two knee replacements, she always sets off the metal detectors and has to undergo a pat-down.
But, she said, a cotton shirt she frequently wears while traveling set off the millimeter-wave machine several times while flying out of Detroit. TSA agents told her it was the shirt’s “kangaroo pocket” similar to those found on sweatshirts that triggered the alarm.
“The last time, they did a full pat-down,” Goldstein said. “The thing that’s ironic to me: I actually like the machine.”
Many travelers, however, also reported false alarms with the X-ray body scanner.
Jason Ritchie, an associate chemistry professor at the University of Mississippi, said he was flying out of Memphis on his way to a conference in August when the operator of the X-ray machine spotted something that required further checking.
The suspicious item: The pockets of his cargo pants, he was told.
“It kind of annoys me when I have to go through the X-ray system because I don’t like to be irradiated unnecessarily,” Ritchie said. “To have to go through that and then be told I also had to get a pat-down was frustrating.”
ProPublica tried to get a handle on the false alarm rate in the United States by commissioning a poll by Harris Interactive. The poll of 2,198 people was conducted online to ensure that those who responded could view images of the machines in addition to reading a description.
Of the 581 people who said they had taken a flight in the past six months, nearly two-thirds, or 367, reported going through a body scanner.
About 11 percent of those scanned said they were patted down anyway despite having nothing on them — the equivalent of a false alarm.
Among this group, the rate of false alarms was slightly higher for the millimeter-wave machine over the X-ray scanner. But Harris Interactive cautioned that because the sample size of people reporting this experience was small, the result cannot be generalized to the population at large.
Possible Solutions
Improving the technology to increase detection but limit false alarms is extremely challenging because of the great variety of body shapes and clothing, said McMakin of the Pacific Northwest National Laboratory.
The machine can be taught to recognize patterns in clothing such as a left breast pocket in men’s dress shirts, he said. But whereas screeners could generally see the outline of an abnormal pocket or buttons in an image, the privacy software eliminates such human discretion.
One option is to combine the millimeter-wave scan with an optical camera to weed out those issues, McMakin said. For example, software could compare the millimeter-wave scan with the photograph to determine if a button or a zipper was causing the alarm. Developers could also increase or decrease the frequency of the waves or improve the shape and location information in the algorithm, he added.
“We’re just at the beginning of where this technology can go,” McMakin said.
Ripp from L-3 said it all comes down to “machine learning.”
Getting the information of what’s normal in order to improve the technology requires many thousands of scans. But because of the privacy outcry, the machines used in airports do not save the images or data from the scans. Without that real-world data, developers have to find other ways to teach the software to distinguish real threats from false ones.
Christian Salewski, a former fellow at ProPublica and a staff writer for the Financial Times Deutschland, reported from Hamburg.
(Copyright 2011 by ProPublica. All Rights Reserved.)
