It is the latest fad in child welfare: Use a computer algorithm that supposedly can predict who will abuse a child. The term commonly used in the field is “predictive analytics.” A more accurate term would be “computerized racial profiling.”
Child welfare is not the only field to embrace predictive analytics. It’s already in use – and proven to be racially biased – in law enforcement, for example. A few members of the New York City Council, including the chair of the council’s Technology Committee, James Vacca, see the danger. They’ve introduced legislation that would, at least, bring a small measure of transparency to how the city uses predictive analytics. Columbia Journalism Review reports that the committee’s hearing on the bill this week was among the best attended in recent history.
The bill comes just in time. The former head of the city’s child welfare agency, Gladys Carrion, expressed doubts about predictive analytics. But she “retired” as Commissioner of New York’s Administration for Children’s Services under pressure after a high-profile child fatality. Her replacement, David Hansell, gives every indication of embracing this new and dangerous fad. That is likely to make worse everything The New York Times found in its recent story about foster care as the new Jane Crow.
The dangers of computerized racial profiling
If predictive analytics worked as well as proponents say it does, Hillary Clinton would be president.
|Remember how the predictive analytics algorithms |
said she would be president?
But, as a Times analysis (published after the election) points out, there is reason for concern about predictive analytics that goes far beyond that one “yuuuge” failure. And those concerns should extend to child welfare.
● ProPublica reports that predictive analytics already has gone terribly wrong in criminal justice, falsely flagging Black defendants as future criminals and underestimating risk if the defendant is white. A new analysis of ProPublica’s data confirmed their findings.
●In child welfare, a New Zealand experiment in predictive analytics touted as a great success wrongly predicted child abuse more than half the time.
● In Los Angeles County, another experiment was hailed as a huge success in spite of a “false positive” rate of more than 95 percent. And that experiment was conducted by the private, for-profit software company that wanted to sell its wares to the county.
● The same company is developing a new approach in Florida. This one targets poor people. It compares birth records to three other databases: child welfare system involvement, public assistance and “mothers who had been involved in the state’s home visiting program.”
So listen-up “at-risk” new mothers: In the world of predictive analytics, the fact that you reached out for help when you needed it and accepted assistance on how to be better parents isn’t a sign of strength – it’s a reason to consider you suspect, and make it more likely that your children will be taken away.
None of this has curbed the enthusiasm of predictive analytics fans. Indeed, Hansell has brought in as a consultant a key backer of the L.A. experiment, the head of the Los Angeles County child welfare agency at the time, Philip Browning.
The campaign for predictive analytics is led largely, though not exclusively, by the field’s worst extremists – those who have been most fanatical about advocating a massive increase in the number of children torn from everyone they know and love and consigned to the chaos of foster care – and also by those most deeply “in denial” when it comes to the problem of racial bias in child welfare.
Some predictive analytics boosters have even argued that “prenatal risk assessments could be used to identify children at risk of maltreatment while still in the womb.” Though these researchers argue that such targeting should be used in order to provide help to the mothers, that’s not how child welfare works in the real world.
“Yes, it’s Big Brother,” said another predictive analytics enthusiast. “But we have to have our eyes open to the potential of this model.”
The real potential of this model was aptly summed up by Yung-Mi Lee, a supervising attorney in the Criminal Defense Practice at Brooklyn Defender Services at the hearing on the New York City bill. Said Lee:
At worst, such tools provide a veneer of color- and class-blind objectivity while exacerbating the racial and economic discrimination and other inequalities in law enforcement practices and criminal and civil penalties.
If anything, the problem is worse in child welfare, where the rest of the process – the records and, in most states, even the court hearings, also are secret.
The New York bill
The New York bill doesn’t do a lot – but it would be a small step forward.
First, it would require that algorithms used “for the purposes of targeting services to persons, imposing penalties upon persons or policing,” be public. That would make it possible for experts to test for bias. It also might eliminate private for-profit companies from pushing their products, since presumably they wouldn’t want their secret formulas made public.
Second, the bill would allow any New Yorker to “plug in” her or his own data and see the result.
So, for example, if the child welfare agency starts using predictive analytics, you could plug in your age, your race, your income level, and whether you’ve reached out for help and see if the agency thinks you’re high risk to abuse your child.
Unfortunately, the bill provides no redress if you find out that you are, indeed, falsely labeled “high risk” because of factors such as race or income (or factors such as whether you’ve changed homes or schools a lot, which are, in reality, measures of race and income).
And transparency alone is not enough. Los Angeles County quietly decided not to move forward with the secret software from a private company – the one with the 95 percent false positive rate. The state of California is now embarking on what it promises will be a transparent algorithm and an open process to develop it. But that alone doesn’t eliminate bias, it only creates the potential to reduce bias.
NCCPR has a detailed discussion of predictive analytics in child welfare in our publication Big Data is Watching You.