A flagship artificial intelligence system designed to predict gun and knife violence in the UK before it happens had serious flaws that made it unusable, local police have admitted. The error led to large drops in accuracy, and the system was ultimately rejected by all of the experts reviewing it for ethical problems.

The prediction system, known as Most Serious Violence (MSV), is part of the UK’s National Data Analytics Solution (NDAS) project. The Home Office has funded NDAS with at least £10 million ($13 million) during the past two years, with the aim to create machine learning systems that can be used across England and Wales.

As a result of the failure of MSV, police have stopped developing the prediction system in its current form. It has never been used for policing operations and has failed to get to a stage where it could be used. However, questions have also been raised around the violence tool’s potential to be biased toward minority groups and whether it would ever be useful for policing.

The MSV tool was designed to predict whether people would commit their first violent offense with a gun or knife in the next two years. People who had already come into contact with the two police forces involved in developing the tool, West Midlands Police and West Yorkshire police, were given risk scores. The higher the score, the more likely they would be to commit one of the crimes.

Historic data about 2.4 million people from the West Midlands database and 1.1 million from West Yorkshire was used in the development of the system, with data being pulled from crime and custody records, intelligence reports, and the Police National computer database.

But as NDAS was starting to “operationalize” the system earlier this year, problems struck. Documents published by the West Midlands’ Police Ethics Committee, which is responsible for scrutinizing NDAS work as well as the force’s own technical developments, reveal that the system contained a coding “flaw” that made it incapable of accurately predicting violence.

“A coding error was found in the definition of the training data set which has rendered the current problem statement of MSV unviable,” aN NDAS briefing published in March says. A spokesperson for NDAS says the error was a data ingestion problem that was discovered during the development process. No more specific information about the flaw has been disclosed. “It has proven unfeasible with data currently available to identify a point of intervention before a person commits their first MSV offense with a gun or knife with any degree of precision,” the NDAS briefing document states.

Before the error was found, NDAS claimed its system had accuracy, or precision levels, of up to 75 percent. Out of 100 people believed to be at high risk of committing serious violence with a gun or knife in the West Midlands, 54 of these people were predicted to carry out one of these crimes. For West Yorkshire, 74 people from 100 were predicted to commit serious violence with a gun or knife. “We now know the actual level of precision is significantly lower,” NDAS said in July.

“Rare events are much harder to predict than common events,” says Melissa Hamilton, a reader in law and criminal justice at the University of Surrey, who is focusing on police use of risk prediction tools. Hamilton wasn’t surprised there were accuracy issues. “While we know that risk tools don’t perform the same in different jurisdictions, I’ve never seen that big of a margin of difference—particularly when you talk about the same country,” Hamilton says, adding the original estimations appeared to be too high, based on other systems she had seen.

As a result of the flaw, NDAS reworked its violence prediction system and its results showed the significant accuracy drop. For serious violence with a gun or knife, the accuracy dropped to between 14 and 19 percent for West Midlands Police and nine to 18 percent for West Yorkshire. These rates were also similar whether the person had committed serious violence before or if it was going to be their first time.