The software, called STRmix, promises to enable police to identify or eliminate suspects based on samples that include DNA from multiple people, such as DNA left on clothing or “touch DNA” left behind when a person touches a surface. But critics call STRmix’s algorithm a “black box” and argue it uses statistics to disguise “guesswork” as settled science.

Introduced by the New Zealand Institute of Environmental Science and Research in 2012, STRmix is now used in 53 forensic laboratories in the US and has been used in more than 120,000 cases worldwide, according to a Feb. 21 press release from STRmix’s public relations firm. It’s among a handful of “probabilistic genotyping” software programs.

Nicole Roehm, who oversees the state’s crime labs as the administrator of the Wisconsin Department of Justice’s Division of Forensic Sciences, told the Cap Times the software is “the general direction that the entire forensic DNA community is heading.

“We need to keep the forensic science work going on in Wisconsin in line with what’s going on around the rest of the world,” Roehm said.

Previously, Wisconsin crime lab analysts testing complex DNA mixtures would calculate a statistic called the combined probability of inclusion, but those results were often inconclusive. “You have so much information there, but it’s so badly mixed together that you couldn’t pull it apart,” Roehm said. 

A 2016 report by the President’s Council of Advisors on Science and Technology found that approach “problematic” and “inappropriately subjective.”

Roehm said STRmix is “much more of a true statistical model representing all possibilities,” offering “the most reasonable explanation” for the DNA in the sample in a way that a single analyst can’t.

Dr. Michael Cox, a professor of biochemistry at UW-Madison whose lab worked with one of the country’s largest crime labs for seven years, agrees. “These profiles can get pretty complicated, so I think it’s sometimes hard to extract all the details by eye,” Cox said. 

“I think having a completely unbiased computer program look at it is a really terrific idea.”

‘A whole new set of problems’

STRmix has been validated by the FBI and the Bureau of Alcohol, Tobacco, Firearms, and Explosives, and STRmix results have been admitted as evidence in 28 U.S. cases, according to the press release, but some in the legal and scientific community have raised questions. 

Last October, the United States Court of Appeals for the Sixth Circuit in Michigan refused to admit STRmix evidence — a decision that has itself been appealed — and multiple legal and reported articles have explored the limitations and possible repercussions of convicting a person based on algorithms and probabilities.

Katherine Judson, executive director of the Madison-based Center for Integrity in Forensic Sciences, said while STRmix might be an improvement over the ways crime labs are currently handling complex mixtures, such analyses remain problematic. 

“There’s still these avenues for human error to be introduced, and there are still problems related to presenting this evidence in court,” Judson said.

Studies show that analysts’ findings can change in response to information police or prosecutors provide, so any tool that offers to eliminate that bias seems promising. 

But, as Katherine Kwong wrote in a 2017 Harvard Journal of Law & Technology article, these technologies designed to solve problems posed by subjective interpretations “have instead opened the door to a whole new set of problems that must be resolved.”

According to the President’s Council report, much is still unknown about STRmix and its competitors, including how the tools’ effectiveness changes with more contributors, smaller samples, or individuals from the same family. The authors asked why different programs have produced different conclusions for the same samples, and they called for independent research into these tools.

The Cap Times attempted to contact STRmix for comment through PR firm Weiss PR, Inc. but did not receive a reply.

Validated by Wisconsin labs

Each lab seeking to implement a new forensic tool must conduct its own internal process to check that the tool produces valid results, and the results of that testing govern when and how the lab may use the new tool. 

Roehm, who previously worked as a DNA analyst, said the Wisconsin laboratories spent a year and a half conducting their validation of STRmix, a process which included testing particularly difficult-to-analyze samples. The labs found the tool valid for samples containing DNA from up to three individuals, as long as the DNA of the individual in question makes up at least 6% of a given sample.

The analyst must still tell the program how many people likely contributed to the sample, based on his or her assessment. Judson considers that an avenue for human subjectivity, but Roehm said DNA analysts can accurately determine the number of contributors in samples of up to four individuals.

Concerns over reliability, transparency

In recent years, forensic techniques that rely on human judgment — among them hair matching, bloodstain pattern analysis and many types of bite mark analysis — have been increasingly criticized as unscientific.

DNA analysis, by contrast, is considered the gold standard in forensic science, engendering trust over the past three decades. But analyzing complex mixtures is a more controversial process, and Judson worries that the public, including jury members, will confuse the two.

“I think what people in the field are really concerned about is that STRmix still involves too much guesswork to be reliable, and that by involving technical software and trained analysts, they’re attempting to make it seem like something other than guesswork,” Judson said.

“But guesswork is still guesswork, no matter how many software programs you use.”

The FBI has approved STRmix for analyses up to five-person samples, but Judson puts little stock in the Bureau’s judgement.

“It’s unfortunate, but the FBI has a history of using forensic sciences later shown to be challengeable, if not outright false,” Judson said, citing the Bureau’s use of hair comparison analysis, which is now known to have led to wrongful convictions.

STRmix evidence has also been contested on the grounds that STRmix’s code is secret, which poses an obstacle to attorneys seeking to challenge the evidence. In 2016, STRmix’s creators authorized attorneys, scientists and expert witnesses to view a “time-limited” version of the software, provided they sign nondisclosure agreements. 

“That’s hardly an open investigation,” Judson said. “You think about anything else in science where you publish your findings, they’re peer-reviewed, people have an opportunity to address them raised concerns, scientists have an opportunity then to alleviate those concerns or go back to the drawing board. That’s not what we’re talking about. 

“We’re saying that this program is going to be used to accuse people of crimes, and we’re going to hope that they have the resources and expertise to mount a challenge to it. You can imagine who gets hurt in that scenario,” Judson said.

Judges’ call

Meanwhile, Wisconsin’s crime labs are moving ahead with STRmix. About a dozen analysts completed a three-month STRmix training at the end of January, and all of the labs’ roughly 50 DNA analysis staff will eventually be trained. Preparing the staff effectively is a “huge undertaking,” Roehm said.

The state’s labs have already begun using the software to analyze DNA in current cases and in some cases where prior tests were inconclusive, Roehm said.

The Division of Forensic Sciences has also begun training Wisconsin prosecutors so that they can understand the results and explain them in courts, Roehm said, and they plan to offer training for defense attorneys as well.

But in order for STRmix test results to be used in Wisconsin courts, the results will have to pass another hurdle: judges. Roehm anticipates each judge seeing STRmix evidence for the first time will hold a “Daubert” hearing to determine whether the evidence meets required scientific standards.

At those hearings, which could start in early 2021, the two sides will likely present the range of arguments for and against this tool, and it will fall to the judges to decide whether to allow the results to be used in court.

If courts accept STRmix, there’s one thing Roehm and Judson can agree on: The tool cuts both ways.

“It’s important for both sides of the criminal justice community,” Roehm said. “We can exclude people that we weren’t able to exclude before.”

Indeed, while DNA evidence is famously used to convict, it’s also been used to exonerate 367 people, according to the Innocence Project.

But Judson, a former defense attorney, said it’s not about which side benefits. “The issue is, when we’re looking at each tool separately, we should be able to trust that it works the way that it’s supposed to, and we should be able to identify … a clear error rate.

 “If you can’t trust the tool, and you can’t trust the process … then it doesn’t matter whether the results are good or good for the defense or good for the prosecution.”