Model Bias

A proposed nationwide class action filed September 7 in the U.S. District Court for the Northern District of Illinois accuses Meta of extracting biometric data from Facebook and Instagram photos to train "NameTag," an unreleased facial recognition system designed for its Ray-Ban and Oakley smart glasses. The complaint, Alvarez et al. v. Meta Platforms, Inc., alleges Meta collected face embeddings, vectors, and templates from user and non-user photos alike, without notice or consent, in violation of Illinois's BIPA and California privacy law. Statutory damages under BIPA run up to $5,000 per intentional violation; the proposed class covers anyone in the U.S. whose image was uploaded to Facebook, Instagram, or fed into a Meta generative AI model.

The suit extends beyond NameTag. Plaintiffs argue that training Emu (initially on 1.1 billion image-text pairs) and its successor Muse Image on face-containing photos caused those models to encode identity-specific facial characteristics in their parameters — and that those representations qualify as biometric identifiers under the law. Meta has said "nothing has shipped to consumers" and that it is "not building a central face database," though WIRED previously reported finding inactive NameTag code in the Meta AI app and evidence the system was designed to retrieve faceprints from Meta servers. Meta previously paid $650 million to settle a BIPA suit over Facebook face tagging and $1.4 billion to resolve a Texas biometric-data case.

Reserve Bank of India Governor Sanjay Malhotra told banks at the FIBAC conference in Mumbai on August 11 to use AI to extend credit to first-time borrowers, gig workers, and small businesses that lack formal financial records. The pitch: models trained on cash flows, GST filings, utility payments, and digital footprints can reach borrowers that manual underwriting never could, and do it cheaply. India has hundreds of millions of underbanked residents and rural literacy below 80%, so the case for AI-powered voice interfaces in local languages is, at least, grounded in a real problem.

Malhotra was not simply handing banks a blank check. He named the risks plainly — opaque decision-making, embedded bias against certain communities and occupations, and systemic fragility if a small cluster of AI providers all ship flawed models at once. His sharpest line was also his clearest rule: "'The model decided' can never be an acceptable answer to a customer, an auditor, or the Reserve Bank." Banks were told to inventory every AI system they run, red-team it before deployment, and keep humans able to explain, intervene, and override any decision that could cause material harm.

Lensa became the most downloaded photo and video app on the iOS store by selling people pictures of themselves. Users upload 10 to 20 selfies and pay $6, or $53.99 for a yearly subscription, to get portraits rendered as anime characters, fairy princesses, and David Bowie-esque space figurines.

The app has also generated nude and cartoonishly sexualised images, including of children, despite a "no nudes" and "adults only" policy. Prisma Labs CEO Andrey Usoltsev told TechCrunch this only happened when the AI was intentionally provoked, which breaches its terms of use. Users of non-Anglo descent reported that Lensa whitened their skin and anglicised their features. Usoltsev said the technology doesn't consciously apply "representation biases", blaming instead "the man-made unfiltered data sourced online". The company's privacy policy says face data is deleted within 24 hours and images are used solely to build the user's own avatars, though the piece notes it is unclear what else an app with that kind of access could pull from a phone without a full audit.

Twitter's automatic image cropper preferred white faces. Over the weekend of 19 September 2020, users ran the feature through a series of side-by-side tests and found it consistently cropped out Black subjects: Barack Obama disappeared from a photo alongside Mitch McConnell, Carl disappeared beside Lenny, and a black labrador disappeared beside a golden one.

The first catch came from PhD student Colin Madland, who posted a picture of himself and a Black colleague to illustrate a separate bias problem in Zoom, where the colleague's face had been erased from a video call. Twitter's preview showed Madland alone. The company said it had tested the model for bias before shipping and "did not find evidence of racial or gender bias in our testing," then promised to open source its analysis so others could replicate it.

England's exams regulator scrapped the algorithm it used to assign A-level and GCSE grades after it downgraded 39% of teacher assessments, prompting protests outside Downing Street and an urgent review by the national statistics regulator.

The warnings ran for months. In May, consultation respondents flagged that "some arbitrary algorithm [is] making standardised adjustments" and that Black and minority ethnic students would be hit hardest. In July, a former Department for Education director general told the education secretary the model would be 75% accurate at best, and external advisers told Ofqual the formula was "volatile" and risked erratic outcomes. MPs warned pupils risked being "systematically disadvantaged by calculated grades" and asked Ofqual to publish the model; Ofqual refused, saying publication would let schools work out their own awarded grades. An adviser who reportedly helped develop the software told the Daily Mail there was "always an understanding there would be winners and losers … there was a very specific point when it became doomed."

A week after the Guardian reported that Microsoft planned to fire the human editors who run MSN.com and replace them with AI, the software illustrated a story about Little Mix singer Jade Thirlwall's reflections on racism with a photo of her bandmate Leigh-Anne Pinnock. Both women are mixed race. Thirlwall, who had recently attended a Black Lives Matter protest in London, called it out on Instagram: "It offends me that you couldn't differentiate the two women of colour out of four members of a group … DO BETTER!" What she could not have known, according to sources at the company, is that the image had been selected by the AI.

Microsoft's fix was to replace the image. The follow-up problem was harder to solve: the robot editor kept selecting coverage of the incident from external news sites and publishing it to MSN.com. Remaining human staff were told to stay alert and delete the article if the system picked it up, and warned that even if they deleted it, the robot editor might overrule them and attempt to publish it again. Staff had already deleted coverage of the Little Mix error after the AI decided readers would be interested.

Amazon's Edinburgh engineering hub spent four years building a recruiting engine that scored job applicants from one to five stars, much as shoppers rate products. By 2015 the team had found the problem. The models were trained on ten years of résumés submitted to the company, most of them from men, and the system had taught itself that male candidates were preferable. It penalized résumés containing the word "women's", as in "women's chess club captain", and downgraded graduates of two all-women's colleges. It also preferred verbs more common on male engineers' résumés, such as "executed" and "captured".

The group built 500 models for specific job functions and locations, each taught roughly 50,000 terms found on past applicants' résumés. Editing out the offending words was no guarantee the machines wouldn't devise other discriminatory ways of sorting candidates, and bad training data meant unqualified people were recommended for all manner of jobs, with results returning "almost at random". Amazon disbanded the team by the start of 2017. It kept a "much watered-down version" of the engine to cull duplicate candidate profiles from databases, and has since formed a new Edinburgh team to try automated employment screening again, this time with a focus on diversity.

The ACLU built a face database out of 25,000 publicly available arrest photos, ran Amazon's Rekognition against public photos of every sitting member of the House and Senate using Amazon's own default match settings, and got 28 matches to people who had been arrested for a crime. The entire test cost $12.33.

Nearly 40 percent of the false matches were people of color, who make up 20 percent of Congress. Six were members of the Congressional Black Caucus, including Rep. John Lewis. Amazon was marketing Rekognition to police at the time, pitching real-time tracking through surveillance cameras and body camera footage, and a sheriff's department in Oregon had already started comparing faces against a mugshot database. The ACLU asked Congress for a moratorium on law enforcement use of face recognition. (ACLU)

Beauty.AI promised objective judging: facial symmetry, wrinkles, no human prejudice. Roughly 6,000 people from more than 100 countries submitted photos, including large groups from India and Africa. Of the 44 winners, nearly all were white, a handful were Asian, and one had dark skin.

The contest was built by a deep learning group called Youth Laboratories and supported by Microsoft. Chief science officer Alex Zhavoronkov said the training data did not include enough minorities, so the algorithm concluded on its own that light skin meant beauty. "If you have not that many people of color within the dataset, then you might actually have biased results," he said. Columbia law professor Bernard Harcourt, who studies predictive policing, called the results "the perfect illustration of the problem." Zhavoronkov said the next round, planned for the fall, would include changes designed to weed out discriminatory results.

Google Photos, launched in May 2015, tags uploaded pictures automatically using its own image-recognition software. On 29 June it labelled a photo of Jacky Alciné and a friend as "gorillas". Alciné posted the screenshot: "Google Photos, y'all fucked up. My friend's not a gorilla."

Yonatan Zunger, Google's chief architect of social, replied to Alciné directly and attributed the error to obscured faces and "different contrast processing needed for different skin tones and lighting". He added that Google "used to have a problem with people (of all races) being tagged as dogs, for similar reasons", and said longer-term fixes were underway covering both linguistics ("words to be careful about in photos of people") and recognition of dark-skinned faces. A spokeswoman told the BBC the company was "appalled and genuinely sorry". Google was not the only one: a month earlier Flickr's auto-tagger had labelled images of black people "ape" and "animal", and pictures of concentration camps "sport" and "jungle gym". Earlier in 2015, Google Maps searches for racist terms returned the White House.

Flickr's new auto-tagging system labelled a portrait of a black man named William with "animal" and "ape". Photos of Dachau concentration camp came back tagged "jungle gym" and "sport". The entrance to Auschwitz got "sport".

Yahoo shipped the feature on 7 May and called it "advanced image recognition technology". A comment thread about it collected almost 2,500 replies, the vast majority negative. Flickr pulled "ape" from the tagging lexicon entirely and left "animal" in place. A spokesperson said the process is completely automated, that no human ever views the photos, and that deleting an incorrect tag teaches the algorithm to perform better in future.