Predictive analytics in child welfare – and elsewhere: A brilliant lesson in understanding algorithms and bias

Are you sufficiently connected to your community?  A simple six-question test might give you the answer.

At least that’s what participants in a forum organized by the NAACP Legal Defense and Educational Fund were told on Saturday.

The multiple-choice questions included: “Of the people you call regularly, do you have their phone numbers memorized?” “Do you know the names of your neighbors?” and “Are you on a first-name basis with your local bodega owner?”

Here’s the full list:

 Later we learned that each time we checked the first box we got 20 points, the second box, ten points, and the third box zero points.

Then we were told that any total under 80 points meant our ties to the community were insufficient.

  

Then we were asked what we thought of the test.  Watch the discussion that starts at 18:24 on this video:

Of course this was not a real test. Instead, as the audience quickly realized, it was a brilliant illustration of the biases that permeate seemingly “objective” algorithms.

And that’s when the designers of the “test,” Toni Smith-Thompson, Daniel Schwarz and Nicole Triplett, of the New York Civil Liberties Union posed this question:

Or, I would add: What could it mean for your life if a government agency wanted to use this test to decide if you’re likely to abuse your child?

No, child welfare isn’t better

Of course those pushing “predictive analytics” algorithms in child welfare say, in effect, we’re better than everybody else, so naturally our algorithms are better, too. (It’s another example of a distressing arrogance in the field, much like when some in child welfare say the field is magically immune from racial bias.)

But if you’re really better, then you don’t have to mislead anyone about your algorithm. With that in mind, let’s review what’s happening in Pittsburgh, home of the Allegheny Family Screening Tool (AFST), the predictive analytics algorithm that supposedly solves the problems of all the other algorithms. 

There are two versions of AFST. The first is used to guide decisions concerning who should be investigated as a possible child abuser.  The second, discussed in more detail below, is even more pernicious.

The algorithms produce a “risk score” – an invisible “scarlet number” that can haunt a child and family for life. The higher the number, the greater the supposed risk.  So consider how the claims of AFST proponents stack up against reality:

They claim: Unlike for-profit firms that sell “black box” algorithms in which the way the risk is calculated is not disclosed, AFST proponents say, in effect: We are transparent, we tell everyone what data points are used in our algorithm.

In fact: They don’t tell us how these data points are weighted, or even if they count for or against a family.  That’s like giving people a recipe that lists the ingredients but doesn’t say how much of each to use.

They claim: An “ethics review” found that our algorithm is so wonderful it would be unethical not to use it.

In fact: There is such an “ethics review.” But one of the reviewers is a faculty colleague and co-author of papers with a designer of the algorithm being evaluated.

Also, the review’s claim that AFST is ethical was predicated in part on it not being applied to every child at birth.  But now, behold the second version of AFST: They’re about to apply it to every child at birth.

They claim: Even though version 2 will be applied to every child at birth, that’s o.k. because participation is voluntary.

In fact: Consent is assumed unless you specifically opt out – and you only get, at most, two chances to do it – if you happen to notice those fleeting opportunities when they arise during the stressful, sleep-deprived first days after your child is born.

They claim: The every-child-at-birth algorithm guards against bias by including only databases that potentially could include anyone regardless of income.

In fact: The weasel-word in that claim is “potentially.” The databases will include use of homeless services, and involvement with juvenile justice and child protective services systems. You don’t see a lot of rich people turning up in those systems.

They claim: The every-child-at-birth algorithm will be used only to target “prevention” services.

In fact: That’s true – for the moment. But once they have the data there is nothing to stop them from changing their minds – say, after a high profile child abuse tragedy.

The simple fact that the proponents of Pittsburgh’s predictive analytics algorithms keep trying to mislead us as they sell their product tells you everything you need to know about the quality of the product.

There's more about predictive analytics in child welfare in our report Big Data is Watching You.