Algorithms gone wild

The medical world has figured out some reasonably solid ways of making transplant algorithms ethical. And my central claim in the academic book is that we can learn from this experience: Some of the practices useful to transplant surgeons might also be valuable in HR departments, or courtrooms, or ... well, other places.

To back this up, my intro chapter will look at six other places where algorithms have gone wild, and where the public or policymakers have struggled to get a handle on the ethics. (At the end of the book, I'll show how each one could profit from methods that are already used in transplant medicine.)

Three of these examples are private sector: a system called HireVue that vets job applicants, the algorithms that set prices for car insurance, and medical algorithm made by a private vendor that warns your pharmacist if it thinks you might be a drug addict. The other three are public sector: the algorithm that matches students to public schools in New York City, the courtroom "risk assessment" systems that increasingly send arrestees to jail or let them go home, and a fraud-detection systems that target welfare recipients.

For each example, I'm writing what law professors call a "squib" – a short summary of the case that sheds light on whatever principle it illustrates. I spent today on the HireVue one, so am sharing that here.

HireVue sets its sights lower

In the pandemic fall of 2020, while working on this book, I taught a class on the ethics of data science for Cornell undergraduates. When it came time for them to write papers about a questionable algorithm, half the class flocked to the same topic: a recruiting system called HireVue, which many of them had encountered that fall while applying for internships. Popular with large employers including JPMorgan and Unilever, the system was a one-way video job interview: students would  record their answers on a webcam to a set list of questions, and HireVue’s software would automatically analyze their “employability.” Many students, after interviewing with the machine, went no further and didn’t get to talk to a human.

HireVue was born as a filing system where human recruiters could store and review video footage of job interviews, but according to a company white paper, in 2015 the company began using “artificial intelligence to transform … video interviews into a … pre-hire assessment.” Their chief psychologist told a Washington Post reporter that the software would automatically decompose each video clip into specific facial movements — apparently, things like a raised eyebrow or a wrinkled nose — which “can make up 29 percent of a person’s score.”

Some students got past the algorithmic bouncer and others didn’t, but none found out what the algorithm had seen in their smile. HireVue claimed to be both old and new, blending cutting-edge AI with a century worth of research in workplace psychology. And it repeatedly referred to itself as “validated,” though it seemed more comfortable making that claim in sales brochures than in scientific papers. In fact, as far as I can tell, before 2020 there was never any peer reviewed, transparent or rigorous analysis of the system’s accuracy. (A related and tricky question is what, exactly, it would even mean to say the system was accurate or not — that might depend upon the job — but the system’s claimed validity was broad if imprecise in scope.) In short, however compelling the private evidence might have been, the details of this system’s operation were something for regulators, job candidates and employers essentially to take on faith. My students pointed out that the system might hurt a candidate with an accent, or the wrong camera, or a different cultural approach to eye contact.

Then, just after my students’ final papers were in, the company made an abrupt and mysterious about-face: it had decided “not to use any visual analysis in [its] pre-hire algorithms going forward.” Why? Because some very recent internal research, which it was not making public, had found that “for the significant majority of jobs and industries, visual analysis has far less correlation to job performance than other elements of our algorithmic assessment.”

Before changing its mind, the company had used algorithms to analyze facial movements and decide on the fates of literally millions of job candidacies. It didn’t quite say in its public statements that these decisions had been tainted with error — the carefully worded blog post left open some chance that facial movements still were positively correlated with job performance, even if “far less correlat[ed]” than other factors were. Still, the change wouldn’t be worth making or announcing unless the old algorithm had made some mistakes. The company was effectively conceding that candidates whose facial movements had most pleased the algorithm had come out ahead — and gotten interviews or jobs that others had lost — even though their smiles weren’t a powerful signal of their ability to succeed on the job.

On the same day of its sudden change, the company released a third party audit of one of its default recruiting models, conducted by the auditing practice of Cathy O’Neil, a wall street quant turned social critic of algorithms who had written the book Weapons of Math Destruction. The audit, which HireVue made public, basically commended the particular slice of the company’s software that O’Neil had examined.

The audit was one of the first of its kind, a first step toward better or different ways of governing high stakes private sector software. And yet, the company’s about-face on the value of analyzing job applicant’s faces — awkward, for a firm whose name sounds like “hire view” — showed that we still have a long way to go in making high stakes software trustworthy.

Meanwhile on the people-are-contextual front…

I was listening to a podcast this morning between Coleman Hughes and Ezra Klein. Ezra had bad grades in school, and struggled to pay attention in the classroom, yet later became a policy wonk and writer. Ezra turned out to have a much greater capacity for learning and writing than his report cards or college GPA might have suggested.) At one point Coleman asks Ezra, how do you square the bad grades with the nerd you've become? And Ezra says:

So the way to the extent I resolve it, the way I resolve it is that context is really important to the success people do or don't have.

It's actually important part of my politics too. I am extremely viscerally, emotionally acquainted with the fact that there are contexts you can put me in, in which I am simply a failure. Like it just doesn't go well for me, And you will look at me and be like, that guy sucks. Like that guy just can't get it together.

And then there are others where I look like a success, but it's the catalytic interaction between my set of strengths and weaknesses. And then the context more than it is anything innate about me. Like I'm not, I don't feel to me like a hugely different person today, but I do have a very different context and that context is allowed different parts of me to be emphasized.

...which made me think, what would a politics of context look like? In other words, suppose we took it to heart that