Predictive analytics in child welfare: If you can’t trust Watson…

Remember Watson?  Watson is that supercomputer that was so good at Jeopardy even Ken Jennings couldn’t beat it.  But it turns out Watson has some limitations.

According to a story in the online news site The Correspondent:

IBM stated in 2016 that Watson would cause a "revolution in healthcare." Instead, several research centres have since cancelled their cooperation with the system because Watson’s recommendations were not only wrong but also dangerous.

Even more striking, because of what it says about how media approach such supposed breakthroughs, is an example to which The Correspondent linked: This story from Health News Review.  Here’s how that story begins:

We often call out overly optimistic news coverage of drugs and devices. But information technology is another healthcare arena where uncritical media narratives can cause harm by raising false hopes and allowing costly and unproven investments to proceed without scrutiny.

The story goes on to cite one gushy news account after another about how Watson would be a great leap forward in treating cancer.

It wasn’t.

So consider: Watson was being asked to help diagnose and design treatments for something that already existed.  It wasn’t even asked to predict much.  And it was dealing in the area of hard science.

Yet we are supposed to believe that algorithms can predict who is going to abuse a child.  And we are supposed to believe that the humans who program these algorithms will magically avoid incorporating into them any human biases.  We are supposed to believe this because, just as happened with Watson and helping to cure cancer, the use of predictive analytics algorithms in child welfare has been the subject of an avalanche of gushy, uncritical news coverage.  (The exception: This story in Wired, an excerpt from Prof. Virginia Eubanks’ book, Automating Inequality.)

In her story for The Correspondent, reporter Sanne Blauw writes:

We’re looking to technology as the sole solution to all our problems. Yet the best solutions could be in a completely different place. Climate change may require systemic change rather than a better algorithm. Teachers and healthcare workers would probably benefit more from a better salary than a robot assistant.

Similarly, child welfare agency caseworkers – and the families they deal with – would benefit more from a better salary than a predictive analytics algorithm.  And child welfare definitely requires systemic change rather than an algorithm.

So here’s your Final Jeopardy answer, Watson:

It doesn’t make children safer, it magnifies human biases, it gathers vast troves of data on mostly poor people without their consent, and the human beings in charge of it will never be able to live up to their promises to strictly limit its use.

The question: What is predictive analytics in child welfare?

UPDATE: Looking at the brighter side, at least Watson wasn't racially biased - which is more than can be said for another recent effort to use predictive analytics in medicine.  

And there’s more about all of this here.

Can child welfare control its “nuclear weapon”? Here’s your chance of a lifetime to find out!

When New York State instituted a lottery in 1967, it came
with all sorts of high-minded promises about how it would
be advertised.  The fate of those promises provides a lesson
concerning whether to believe similar promises about predictive
analytics in child welfare. (Photo by Reuben Whitehouse)

Predictive analytics is the nuclear weapon of child welfare.  Vast amounts of data are taken from people – especially poor people – without their consent (like what Facebook does, only worse). Then if someone alleges that one of those people has committed child abuse or neglect, a secret, or perhaps only semi-secret, algorithm coughs up a risk score. That score is an invisible scarlet number that can brand not only parents, but their children, for life.

As Author Virginia Eubanks explains in her book, Automating Inequality, rather than counteracting the racial and class biases of the human beings who run child welfare systems, it magnifies those biases. She calls it “poverty profiling.” And ProPublica has documented how this has played out in criminal justice.

So what is the response from proponents of analytics in child welfare – a field that is super secretive with no real accountability, due process, or checks and balances?  Endless promises of self-restraint.

Sometimes they say: “We’ll only use it to target prevention programs.”  But we already know how to target prevention programs without an algorithm: Just put them where the poor people are, since the overwhelming majority of cases involve “neglect” and child welfare systems routinely label poverty as neglect.

Or they’ll say: “Child abuse hotline operators will know the “risk score” but we won’t even tell the people who actually go out to investigate the allegation.”  But whoever is going out to investigate knows that if they’re told to get out there in a hurry it’s probably because the risk score was high. So whether they’re told or not, they know.

Or they’ll say: “We’ll never, ever use the score to decide whether to remove a child from the home.”  But again, the caseworker knows (whether explicitly told or not) when the algorithm has rated a case high risk – and they can’t unknow it when the time comes to decide whether to remove the child.

Or they’ll say: “Even if we get our fondest wish and get to slap a ‘risk score’ on every child at birth (and make no mistake, for some in the field, it is their fondest wish) we’ll only use it for prevention.  But – well, see all the problems cited above.

The limits of high-minded promises 

But there’s an even bigger problem with all these high-minded promises. What happens as soon as there’s pressure to be less high-minded?

The amount of pressure needed to get politicians to abandon their principles can be remarkably low – as is illustrated by the story of the New York State Lottery.  Yes, the Lottery.

New York State was among the first in the modern era to institute a lottery, in 1967.  It took an amendment to the State Constitution – so there were lots of high-minded promises to allay concerns of those who feared it would encourage compulsive gambling or encourage those least able to afford it to waste their money.

The key selling point, it was promised, would be an appeal not to greed but to generosity.  Advertising would emphasize that lottery proceeds would be used to help fund public education.  So the first lottery slogan was "Your Chance of a Lifetime to Help Education."  I grew up in New York and I recall an early print ad that said “The New York State Lottery: It’s not the money; it’s the principal. And the teachers. And the students.”

There was just one problem. Not enough people were buying lottery tickets.  Sales were way below projections.  So, by the 1980s, the lottery took a different approach that might best be called, it’s not the principals and the teachers and the students – it’s the money! Money! Money!  Have a look:

Yes, the Lottery still sometimes produces commercials that take the high road, but this is the dominant theme.

If all it takes is revenue falling short of projections to prompt this abandonment of principle (and principals), imagine what would happen in a field where the stakes are a lot higher.

Imagine this scenario:  A child “known to the system” has died.  The media have found out the name of the caseworker who mistakenly thought the home was safe. After being attacked in news accounts and/or by politicians, she comes forward to tell her story.

Choking back tears, she says: “My bosses had an algorithm that told them this family was high-risk. But they never told me. Of course, if only I’d known I never would have left that child there.”

What are the odds that the leader of the child protective services agency would stick to the policy of using predictive analytics with only the utmost restraint?  Even if s/he wanted to, what are the odds that the political leadership in the state or county would allow such restraint to continue?

I’d say you’ve got a better chance of winning the lottery.

Predictive analytics in Pittsburgh child welfare: No poverty, no profile?

Pittsburgh's predictive analytics algorithm labels parents - and children - with a
 "risk score" that amounts to an invisible "scarlet number" that may haunt them for life.

Pittsburgh is the home of the nation’s most advanced use of “predictive analytics” in child welfare.  In Pittsburgh, an algorithm is used to screen calls alleging child abuse and determine if a case is high risk.  The algorithm produces a “risk score” – a secret “scarlet number” between 1 and 20. The higher the number the greater the supposed risk.

So now let us consider a recent incident in Pittsburgh – and what the algorithm might do.

A man storms into an office.  His 12-year-old daughter is with him. The man appears intoxicated.   The man screams and yells and pounds his fists against a bulletin board. He demands to have his picture taken.

He forcefully grabs his daughter’s forearm, pulling her into the picture as she tries her best to pull away from him.  She screams “Please, please Daddy, no!” multiple times. And multiple times he yanks on her arm, trying to pull her to his side so a photo could be taken of both of them.  He yells at his daughter and repeatedly jabs his finger in her shoulder.

The daughter is crying hysterically and red-faced.  The father rips a cell phone out of her hand because he thought she was trying to call her mother.

As one eyewitness said:

I was extremely concerned for his daughter‘s safety, and I actually noticed that my heart was racing. …  Having to watch as [the father] terrorized his teenage daughter — with his hair disheveled and his face twisted — was something I’m never going to forget.

What would AFST do?

I don’t know if anyone called Pennsylvania’s child abuse hotline to report the incident.  But if anyone did, and if the call were then referred to Allegheny County Children and Youth Services (CYS), the name of the father would be run through the Allegheny Family Screening Tool (AFST), the county’s vastly overhyped predictive analytics algorithm.  And the odds are that this father’s risk score would be very, very – low.

Why?  Because the father in this case is John Robinson Block, publisher of the Pittsburgh Post-Gazette.  The alleged outburst occurred in the Post-Gazette newsroom.  The account above is taken directly from eyewitness accounts posted on the website of the Newspaper Guild of Pittsburgh.

Block Communications disputes these accounts. According to The New York Times:

In a statement on Thursday, Block Communications disputed the employees’ accounts, saying that Mr. Block had only “expressed his frustration” to employees “about several issues of concern to him.” The company said it provided a safe work environment. 

“We have conducted a review of all information available, and we disagree with the characterization of Saturday evening’s events as expressed by the Newspaper Guild,” the statement said. Mr. Block “expresses his sincere regrets over his conduct that evening and did not intend his actions to upset anyone,” it added.

But, of course, child protective services agencies urge people to report even their slightest suspicions that something they’ve seen might be child abuse or neglect. 

Were John Robinson Block reported, AFST would not cough up a message that says: “Hey, this guy’s a bigshot publisher, better leave him alone!” but as a practical matter, the algorithm may have a similar effect.

As Prof. Virginia Eubanks explains in her book Automating Inequality, 25 percent of the variables in AFST are direct measures of poverty. Another 25 percent measure interaction with the child welfare and juvenile justice systems themselves.

As Eubanks explains:

Because the model confuses parenting while poor with poor parenting, the AFST views parents who reach out to public programs as risks to their children.

Because these are public benefits, such as SNAP (formerly foodstamps), TANF (Temporary Assistance for Needy Families) and Medicaid, the data are collected automatically by the county.

But odds are John Robinson Block has never applied for any of these programs, so his risk score is likely to be lower.

And if John Robinson Block has ever had any personal problems that might bring him to the attention of, say, health professionals, he would have been able to get the best private care – so nothing is going to go into a public database that could be scoured by AFST and further raise the risk score. 

Eubanks calls AFST “poverty profiling.”  But is there a corollary: No poverty, no profile?

In her book, Eubanks documents impoverished families caught in the Allegheny County CYS net based on allegations far less serious than what those eyewitnesses say occurred in the Post-Gazette newsroom.

It’s possible that the whole incident wouldn’t be in Allegheny County’s jurisdiction anyway. I don’t know if Block even lives in the Pittsburgh area. The family media company is headquartered in Toledo, Ohio.

But if the allegations ever do reach screeners in Allegheny County, workers may be too busy checking out “high risk” families whose poverty has been confused with “neglect” to take them seriously.

On one level that might be good for Block’s daughter.  As I noted in this column for Youth Today:

The algorithm visits the sins of the parents, real or imagined, upon the children. Eubanks cites a family that was victimized by repeated false reports. When the child the county supposedly was “protecting” grows up, if she herself becomes the target of a child abuse report she will bear a higher scarlet number — because her own parents supposedly were neglectful. So her children will be at greater risk of enduring the enormous harm of needless foster care.

In contrast, if the parent’s “scarlet number” is low, the child’s will be as well.

No this does not mean we should do more spying on rich people

I can hear America’s latter-day “child savers” now: “This just proves we need even more spying!” they’ll say.  “Make butlers and chauffeurs mandatory reporters of child abuse!” In fact, Erin Dalton, deputy director of the Allegheny County Department of Human Services, has already said something like that, telling Eubanks: “We really hope to get private insurance data. We’d love to have it.”  

As I’ve noted before, Dalton also is the one who sent an email talking about stamping a scarlet number on every child born in the county – at birth. She’s also gone out of her way to  minimize the harm of foster care.

The child savers also will look at a case such as this and say: “Just because we can’t find out nearly as much about rich people doesn’t mean we should let poor people get away with child abuse!”

That, of course, misses the point. The fact that, if the eyewitness accounts are correct, and if he were reported to Allegheny County CYS,  John Robinson Block probably would wind up with a low AFST risk score does more than illustrate what doesn’t get noticed. It also illustrates what does get noticed.  It illustrates that AFST doesn’t predict child abuse – it predicts poverty, and then confuses that poverty with neglect.

Foster care apologists shouldn’t have nuclear weapons

The likely next leader of the child welfare system in Pittsburgh co-authored an appalling defense of foster care. She’s also considering stamping a “scarlet-number” predictive analytics risk score on children at birth.

Allegheny County's predictive analytics algorithm operates
like an invisible "scarlet number" that can harm a child for life.

I’ve written often about the dangers of the latest fad sweeping through child welfare, “predictive analytics.”  The idea is to use an algorithm to predict which parents supposedly are likely to abuse their children. Proponents say it reduces human bias. In fact, it magnifies human bias and gives it a false veneer of objectivity.  It is the nuclear weapon of child welfare.

So it’s no wonder that the most prominent proponents of predictive analytics also are those who are most fanatical about wanting to tear apart more families – and often those most deeply in denial about the problem of racism in child welfare. The predictive analytics cheerleading squad is led by people such as Elizabeth Bartholet and Richard Gelles, and Gelles' principal disciple, Naomi Schafer Riley.

Indeed, it is very hard to find anyone who supports this kind of computerized racial profiling who is both a real advocate of family preservation and does not run a child welfare system.

For those who do run such systems, the temptation can be irresistible.  That brings us to Pittsburgh and surrounding Allegheny County, Pa. That jurisdiction is the only one I know of where predictive analytics is up and running. (Similar efforts in Los Angeles and Illinois failed spectacularly.) The Pittsburgh experiment has been the subject of numerous gushy tributes from people like, well, Naomi Schaefer Riley – and one real critique, a chapter in Virginia Eubanks’ book, Automating Inequality, excerpted in Wired.

In Pittsburgh, when a call alleges abuse or neglect, an algorithm known as the Allegheny Family Screening Tool (AFST) mines a vast trove of data on the accused and coughs up a “risk score” for the child.  Like an invisible scarlet number, the child will wear that “risk score” for life – even if the original report on the parents was false.

So when the child  grows up, if she herself becomes the target of a child abuse report, that high scarlet number from her childhood will count against her, making her look like a “higher-risk” parent – because supposedly, she was at “high risk” as a child.

The argument made by backers AFST boils down to this: Our system is run by really good people. Marc Cherna, the longtime director of the county Department of Human Services, has a solid track record for curbing needless foster care.  He has promised to use predictive analytics only in limited and responsible ways. In other words, you can trust him with nukes.

To which I and others have replied:

What about Cherna’s successor, and his successor’s successor? Any system that depends for success on the benevolence of a single leader with near-absolute power is too dangerous for a free society. Most of those pushing for the use of systems like AFST are nothing like Marc Cherna. On the contrary, they tend to be those most enthused about taking away more children and using algorithms to accomplish it.

Cherna’s likely successor is his deputy, Erin Dalton. She runs DHS’ Office of Data Analysis, Research and Evaluation. 

Predictive analytics is the nuclear weapon of
child welfare - and child welfare can't control its nukes

As Eubanks reveals in her book, Dalton is the author of an email disclosing that the county is considering introducing “a second predictive model … [that] would be run on a daily or weekly basis on all babies born in Allegheny County the prior day or week.” Such a model already exists — indeed it’s one of the models the designers of AFST proposed to the county in the first place.

In other words, Dalton is seriously considering a plan to stamp that scarlet number on every child in the county – at birth.  Once again, the response is assurances that, were this to happen, it would only be used to target prevention, not removal.

But now there is new reason to question such reassurances, and, indeed, any confidence that Dalton will act with restraint.  While I certainly wouldn’t call her enthusiastic about taking away children, she recently has shown herself to be far too sanguine about the harmful effects of child removal.

That is clear from a commentary she co-authored for the journal Pediatrics. (Other co-authors include Dr. Rachel Berger who runs the "Child Advocacy Center" at the University of Pittsburgh Medical Center.) The commentary puts her firmly in the camp of foster-care apologists, the people who desperately look for scholarly straws to grasp in order to refute the mountain of evidence that foster care often does enormous harm to the children it is meant to help.  I expect this from Naomi Schaefer Riley.  I did not expect it from Erin Dalton.

Dalton’s bizarre commentary is an attack on an innocuous little study. The study demonstrated that teenage mothers who give birth while already in foster care are far more likely to have the infants taken from them than teenage mothers who are not in foster care.  Half the teen mothers in foster care had their own children placed by age two.

The study’s conclusion is hardly radical: “More and better services are required to support these mothers and to keep mothers and children together wherever possible.” 

Dalton & Co. have several complaints about the study.  The study looked at just one jurisdiction and it’s in Canada, no less – the province of Manitoba - so it may not be representative.  Many of the children were taken at birth, they write, so maybe they were placed in the same foster home as their mothers.  But the authors of the study believe this was the case for only “a few” of the children – and the gaps they found in placement rates persist all the way to age 2.

But the most alarming part of the critique from Dalton and her co-authors is this:

The outcome measure selected for this study (placement of the infant into foster care) is not the most important outcome for children and young mothers. Avoiding unnecessary foster care placement is a worthy goal, but placement of an infant, a young child, or an adolescent mother in foster care is not a bad outcome per se.

That is dangerously wrong. Foster care is, in fact, a bad outcome per se, and everyone in child welfare is supposed to know it.

Foster care may, in some limited circumstances, be a less bad outcome than leaving the child in her or his own home. For that very reason, there are a limited number of cases in which foster care placement is essential. 

But it is still a bad outcome.  As I discussed in my previous post about foster care apologists, foster care sometimes may be the least detrimental alternative – a concept that should, by now, be Social Work 101.  The fact that a child welfare leader who has child welfare’s equivalent of the nuclear codes doesn’t get this is deeply disturbing.

And it gets worse.  Like Riley, Dalton and her colleagues desperately seek something to refute the massive, comprehensive studies showing that in typical cases, children left in their own homes fare better even than comparably-maltreated children placed in foster care.  The studies, by MIT Professor Joseph Doyle, looked at what actually happened to more than 15,000 children, following them all the way into their late teen years and sometimes into young adulthood.

A reminder of what MIT Prof. Joseph Doyle's massive studies actually found

Dalton & Co. ignore those studies. Instead, they write:

The authors of several longitudinal studies suggest that under certain circumstances, foster care may result in better long-term outcomes than the outcomes of remaining with biological parents.

Leaving aside the use of the demeaning, dehumanizing term “biological parents” – suggesting people who are no more important to a child than a test tube, and leaving aside the fact that three studies barely qualify as “several” – the “longitudinal studies” cited are extremely weak.

As with the studies cited by Riley they depend not on what actually happened to the young people, as the Doyle studies do, but on subjective evaluations of children’s behavior, including evaluations by caretakers – creating significant potential for bias, or simply honest error.

One of the longitudinal studies didn’t have much longitude – it measured changes in three groups of children after only six months – and the three groups had a total of only 92 children.  The study authors themselves call it a “short-term follow-up,” yet Dalton and her co-authors try to use it to justify claims about “long-term outcomes.” 

Another study, again using subjective evaluations, involved only 30 children – from Israel. So now a study from Israel is cited by the same people who question the validity of relying on a study from Canada.

The third study also was a subjective assessment. It appears that the assessment took place only once, so it is unclear whether this study was “longitudinal” at all.  This study found that “maltreated children who remain with their birth parents have mental health problems at the same rate as maltreated children who are placed.” [Emphasis added.] Not exactly a ringing endorsement of foster care.

That is the “evidence” Dalton and her co-authors cite to justify this claim:

The assumption that reducing foster care placements always improves outcomes is not necessarily true and may be used to support policies that are not in the best interests of children.

No one claims that reducing foster care placements always improves outcomes. But it almost always   And that makes it entirely reasonable to worry that outcomes are worse for children of teen mothers when those children are placed in foster care.

does.

As for “policies that are not in the best interests of children” what policies exactly do Dalton and her coauthors have in mind?  They never say.  The study they criticize calls only for “more and better services.” Surely they don’t want fewer and worse services.

And the very use of the phrase “best interests of the children” is another indication that child welfare in general and an agency Dalton is likely to run in particular are not ready for something as powerful and easy to misuse as predictive analytics.

I discuss why this seemingly benevolent and inarguable phrase is so harmful in this post, dealing with Maine’s governor, Paul LePage a kind of Donald Trump mini-me. Suffice it to say here that it is a phrase filled with hubris. It gives free reign to the biases of middle-class professionals – and the algorithms they create. The alternative construct, least detrimental alternative, was suggested in part for that very reason.

I expect no better from Paul LePage.  I used to expect far better from the system in Allegheny County, Pa.

The nuclear weapon of predictive analytics is far too dangerous to entrust to the field of child welfare. What we need to demand from child welfare is irreversible, verifiable denuclearization.

Predictive analytics in child welfare: The harm to children when “the stuff stays in the system.”

Marc Cherna is was once one of the best human services leaders in America. But even he shouldn't have the power to be the Mark Zuckerberg of child welfare.

Today, across America and much of the world, the big story will be Facebook CEO Mark Zuckerberg testifying before Congress about how personal data from millions of Americans wound up in the hands of Cambridge Analytica. Although the data breach is outrageous, at least those data were originally uploaded voluntarily – Facebook users have the right to not share their data in the first place.

In Pittsburgh, Pa. poor people have NO. SUCH. CHOICE. They are forced to surrender their data.  And their data can be used to decide whether to take away their children. I’ve written about the implications here and here.  Another example comes courtesy of a Pennsylvania dentist:

Last week, I published a post about a dentist in Pennsylvania who sent threatening form letters to some of his patients. The patients had dared to not schedule follow-up appointments when the dentist thought they should. In the case which brought this to public attention, the patient didn’t like the dental practice and had made clear her intention to go elsewhere.

The letters threaten to report patients who don’t schedule follow up appointments to child protective services.  According to at least one news account, the dentist acknowledges following through on the threat 17 times last year.

The earlier post discusses the potentially devastating consequences for children. If the report is “screened in” – as is likely because it came from a medical professional – it means, at a minimum, a highly intrusive investigation that could do lasting emotional harm to the children.  That harm can’t be undone if the child welfare agency realizes the report was false.

The mere existence of a false report in a child welfare agency file can increase the chances that, if there’s another false report, the new report will be wrongly substantiated – because of a bizarre notion in child welfare that enough false reports are bound to equal a true report. This increases the odds that the children will be consigned to the chaos of foster care.

And, of course, all those false reports steal time caseworkers should be spending finding children in real danger.

The only good news here is that this dentist practices in eastern Pennsylvania.  At least in that part of the state a child abuse “hotline” operator deciding if a case should be “screened-in” can check the file and, seeing a previous allegation based solely on a missed dental appointment, might realize how absurd it was.

Were this dentist at the other end of the state, in Allegheny County (metropolitan Pittsburgh) it could be far worse.

Automating absurdity

That’s because Allegheny County is home to the nation’s most advanced experiment in using “predictive analytics” to decide when to investigate if a child is in danger of being abused or neglected. 

Whenever the county receives a report alleging that a child is being abused or neglected, an algorithm known as the Allegheny Family Screening Tool (AFST) uses more than 100 different data points to spit out a secret “risk score” between 1 and 20 -- an invisible “scarlet number” that tells the county how likely it is that the child is, in fact being abused or neglected or is at risk of abuse or neglect.  The higher the number the more likely the report will be “screened in” and investigators will be sent out.

Though the investigators don’t know the risk score, they do know that a high risk score is why they are being sent out in the first place.

Prof. Virginia Eubanks offers a devastating critique of AFST in her book, Automating Inequality. Part of that chapter is excerpted in Wired magazine. I discussed her findings in detail in Youth Today and I discussed the ethically-challenged “ethics review” used to justify AFST on this blog, so I would repeat that overall critique here.

But the case of the disgruntled dentist prompts me to focus on one particular piece of the Allegheny algorithm: The mere fact that a previous report exists – regardless of how stupid that report may have been – raises the risk score.  No human being intervenes first to see if the report had any legitimacy.

In fact, it appears that the Allegheny County algorithm even counts previous reports that were considered so absurd they were screened out with no investigation at all. 

So suppose, hypothetically, an Allegheny County dentist reported someone just for missing a follow-up appointment.  This was considered too absurd even to investigate.  A few months later someone else calls the child abuse hotline about the same family.  The existence of that previous, uninvestigated report from the dentist raises the risk score.  So now, the child has a higher scarlet number.

Making it even worse: In the Allegheny algorithm still another factor increasing the risk score is if a report, no matter how absurd, was made by a medical professional – such as a dentist.

And Cherna is pretty fanatical about keeping and using data – regardless of the data’s reliability.  This is clear in what he told Prof. Eubanks about a related issue: reports that are legally allowed to be kept for far longer, those in which caseworkers “substantiate” the allegation. In Pennsylvania, as in most states, that means only that the caseworker decides it is slightly more likely than not that abuse or neglect occurred.

In cases alleging actual abuse, there is a long, almost impossible appeals process.  And in a bizarre twist of Pennsylvania law, in less serious cases there is no appeals mechanism at all.  In such cases, the county keeps a record of the case until the child who was the subject of the report turns 23. 

This is where we find out how Marc Cherna feels about keeping junk reports and using them in his algorithm.  He told Eubanks: “The stuff stays in the system.” And, of course, Cherna said what those who hold onto junk reports of child abuse always say. In effect, enough false reports are bound to equal a true report. Said Cherna: “A lot of times where there’s smoke there’s fire.”

But a lot more often there’s someone who’s just blowing smoke.  So let’s just be grateful that a certain dentist hasn’t opened a branch office in Pittsburgh.

And, as we watch what's happening in Congress today, let’s also remember one thing more.  Marc Cherna is now the Mark Zuckerberg of child welfare. Both Zuckerberg and Cherna amass huge quantities of data. Then they decide what will happen to those data.  There are two key differences: Marc Cherna isn’t doing it to make money. In fact both his intentions, and his track record as a child welfare leader are excellent.  On the other hand, Facebook can’t use data to take away your children. Marc Cherna’s agency can.

UPDATE: 11:45 am: In an earlier version of this post, I asked whether inclusion of certain elements in AFST violated Pennsylvania law concerning expungement of records involving unfounded reports. This was based on a list of factors included in AFST. I noted that I had e-mailed Cherna and his deputy Erin Dalton on April 5 and received no response.  I have just received a response from Cherna, in which he makes clear that, in fact, AFST does NOT include any information that legally should be expunged.  Therefore, I have deleted that portion of the original post.

Predictive analytics in Pittsburgh child welfare: Was the “ethics review” of Allegheny County’s “scarlet number” algorithm ethical?

Allegheny County's use of predictive analytics in child welfare
generates the equivalent of a "scarlet number" that can mark
a child, and even that child's children.

Imagine you’re on a baseball team. The game is about to start when you find out that the home plate umpire moonlights as a volunteer coach for the other team.  He even co-authored a handbook with the manager of the other team.

That umpire still may be perfectly capable of objectively calling balls and strikes.  But would you be comfortable relying on him to make those calls?

Anyone who would say no should be deeply uncomfortable about one of the key elements in how Allegheny County (Pittsburgh) Pa. sold its use of “predictive analytics” to decide which families are investigated as alleged child abusers – and possibly, in the near future, to decide much more.

The Allegheny County model is known as the Allegheny Family Screening Tool (AFST). In effect, it stamps children who are the subject of reports alleging child maltreatment with an invisible “scarlet number” that supposedly measures how likely they are to be abused or neglected. I discussed the dangers of AFST in this column for Youth Today.  Here I’d like to focus on one element that has been crucial to selling AFST.  Over and over stories proclaiming how wonderful it is cite a so-called “ethics review.”  An op-ed in the Pittsburgh Post Gazette declares:

 The ethical assessment determined that the tool was so much more accurate than relying solely on human analysis that declining to use it would be unethical.

Similarly, a New York Times Magazine story, which I’ve discussed before on this blog, declares:

 Marc Cherna, who as director of Allegheny County’s Department of Human Services … had an independent ethics review conducted of the predictive-analytics program before it began. It concluded not only that implementing the program was ethical, but also that not using it might be unethical.

But “independent” is a stretch.

The review was conducted by Prof. Eileen Gambrill of the University of California – Berkeley and and Prof. Tim Dare of the University of Auckland in New Zealand. New Zealand? Seems like a long way to go to find an ethics reviewer.  Unless, of course, his selection has something to do with this: One of the designers of the predictive analytics model used in Allegheny County is Prof. Rhema Vaithianathan of the University of Auckland in New Zealand.

And these two don’t just pass each other in the hallway.  Much like that hypothetical manager and umpire, Prof. Dare and Prof. Vaithianathan co-authored papers.

That doesn’t mean Prof. Dare can’t be objective.  But if Allegheny County is going to commission an ethics review and tout it as independent, surely among all the world’s academicians the county could have found two with  no ties to the authors of the model being reviewed, thereby ensuring there would not be even the appearance of conflict-of-interest.

Unless, of course, the county was afraid that such a review wouldn’t tell county officials what they wanted to hear.

That’s probably why, while looking for scholars in New Zealand, they decided not to ask Prof. Emily Keddell of the University of Otago.  She’s already done a review of some of Prof. Vaithianathan’s work – and it’s not nearly as favorable as the one written by Vaithianathan’s co-author.

As for the review itself, it’s a startlingly superficial document, a nine-page once-over-lightly that starts on page 44 of this document.  Citations are few and far between – and they are limited to papers written by either the designers of AFST or Prof. Dare himself.

A key caveat

But even so, as I noted in the Youth Today column, the review includes one important caveat:

AFST is said to be ethical in part because it is used only after a call to a child protective services hotline alleging abuse and neglect.  The review specifically questioned whether AFST would be ethical if applied to all children at birth.  As the review states:

[The issue of informed consent] is one of a number of points at which we think that it is ethically significant that the AFST will provide risk assessment in response to a call to the call center, rather than at the birth of every child. In the latter case there is no independent reason to think there are grounds to override default assumptions around consent. The fact there has been a call, however, provides at least some grounds to think that further inquiry is warranted in a particular case. [Emphasis added.]

But that might not be the case for long. As I noted in my column for Youth Today, in her brilliant book, Automating Inequality, Prof. Virginia Eubanks reports

that the county is, at a minimum, considering introducing “’a second predictive model …[that] would be run on a daily or weekly basis on all babies born in Allegheny County the prior day or week,’” according to a September 2017 email from [Cherna’s deputy, Erin] Dalton.”  Such a model already exists – indeed it’s one of the models the designers of AFST proposed to the county in the first place.

So if the county starts branding every infant with a scarlet number at birth, a number that will even affect the number assigned to their children and grandchildren, is that model inherently unethical? 

I’ve known Marc Cherna for more than 20 years. When I lived in Allegheny County I served on a  screening committee that unanimously recommended him, and one other candidate, as finalists for the job of running the county child welfare system. We were right. He has an outstanding record for safely keeping families together. 

I’ve included Pittsburgh on NCCPR’s list of ways to do child welfare right, and referred journalists from everywhere from CNN, in 2002, to the Arizona Daily Star, just this year, to Pittsburgh to examine the county’s success. (Cherna still regularly quotes what I told CNN all those years ago.)  When Cherna says he wants to use analytics in the right ways for the right reasons, I believe him. 

But that’s not enough. And touting a stacked-deck ethics review is particularly disappointing. Again, as I wrote in Youth Today:

Cherna promises that the scarlet numbers under any such system will be used only to find the families most in need of help. … But  what about Cherna’s successor, and his successor’s successor?  Any system that depends for success on the benevolence of a single leader with near absolute power is too dangerous for a free society.

NCCPR’s full analysis of the role of predictive analytics in child welfare is available here.