3.1 - Bias in automated selection

Bias, implicit or explicit, has always been present in decisions over prioritization and selection of information (“The lost meaning of ‘objectivity’ - American Press Institute,” n.d.). As we saw in Chapter 1, explicit bias is understood (and sometimes expected) from certain types of journalistic outlets. For example, a newspaper with clear political affiliations will by definition prioritize those types of information that benefit the viewpoint of its political heading, and that of its readership (Groseclose & Milyo, 2005). Bias, especially political bias, has in fact an aggregating groupthink function that is sometimes welcome and expected from politically affiliated newspapers and portals. Additionally, this type of bias is relatively well understood and expected from the readership: it is a known quantity, and the individual’s capacity to parse information takes into account the expected bias when deciding on the veracity and importance of information that is being presented. Critical understanding of the fact that journalists are human beings with their opinions and agendas is part of our individual sense-making toolbox, albeit in different measures depending on the individual’s level of education, cultural and geographic background.

With algorithms, the situation is radically different — and yet the same. This paper posits that, while algorithmic selection of information holds equal risk of presenting implicit and explicit bias as the traditional gatekeeping structure (although in different shapes, explained below), the reader’s expectation and critical parsing of bias from a machine-based system is less well-developed. The public holds an expectation of agnosticity from algorithmically parsed information that is simply not there.

Provenance of bias in algorithms

More than mere tools, algorithms are also stabilizers of trust, practical and symbolic assurances that their evaluations are fair and accurate, free from subjectivity, error, or attempted influence. But, though algorithms may appear to be automatic and untarnished by the interventions of their providers, this is a carefully crafted fiction. (Gillespie, 2014, p. 13)

By delegating gatekeeping to algorithms, society has also enacted a transfer of power: from the journalistic power structure of the past, to algorithm developers and owners.Discussion of merit and positive or negative qualities of this new power structure are not relevant to the fundamental point: that this transfer of power happened, and it is increasingly influential in deciding the type of content we receive. From a bias perspective, this transfer of power creates two new aspects of bias provenance: implicit and explicit.

When considering implicit bias, i.e. unconscious manipulation of algorithmic results when making certain preferential choices rather than others on how those algorithms will parse incoming information, the first point on the agenda is the geographic, societal and cultural provenance of algorithm developers and owners. For developers, the numbers are in: Stack Overflow, one of the biggest and most influential communities for programmers and developers with over thirty million monthly users, ran a demographic survey in 2015 of its users (“Stack Overflow Developer Survey 2015,” n.d.). More than twenty-six thousand people from over a hundred and fifty countries participated in the survey, and its results can be seen as a strongly representative snapshot of the average developer. According to Stack Overflow, the average developer is almost exclusively male (more than 92% of respondents), between 20 and 35 years old (more than 70% of respondents), and resides overwhelmingly in Northern America, Europe or India. This survey paints a bleak, monothematic and exclusivist picture: it effectively confirms that automated processes for parsing information are created by a very limited subset of the population — whose algorithms influence the population of the entire globe. It is important to mention that majority rule and minority representation is a societal problem in developer culture as much as it is in other groups — hence, Stack Overflow survey results necessarily skew the numbers to over-represent the majority group. Even taking this caveat into account, the study’s sample case is large enough to consider it a decent approximation of the ecosystem, even when taking into account the underrepresentation of minority voices (arguably, this caveat also corroborates the main point). When considering implicit cognitive bias, in particular the mechanisms of confirmation bias and framing, it is safe to assume that developers will not be any more immune to its effects than other social groups. When linking the concept of bias to the abovementioned search for efficiency in algorithmic processes, it is likewise safe to assume that efficiency itself is not a universal concept. Rather, human beings act upon perceived efficiency - and that perception of efficiency will be influenced and manipulated by implicit bias.

We can stay a while longer in the analysis of implicit bias and talk about algorithm owners, the second group of relevance for this section. Owners in this case are the people and organisations that invest resources into building specific algorithmic processes for the purpose of improving their services to their customers. They are also usually the ones hiring developers to develop algorithms. As an example, we can take social media giants like Facebook and Twitter, or search engine behemoths like Google. Unsurprisingly, there is a very similar lack of race, gender and geographical diversity - the average CEO is overwhelmingly white, male, and US-born (Stacy Jones, 2015). Consequently, this means that not only the creators of algorithms will bring a very specific and commonly shared set of implicit biases into developing algorithms — the people commissioning those same algorithms are usually bringing the same biases when deciding on a higher level which algorithms to implement, i.e. how to improve the way their companies should filter and process information.

The origins of explicit bias, on the other hand, have some similarities to the demographic setup of developers and owners, but are also influenced by market decisions - additionally, explicit biases usually stem from owners. The choice to under- or over-represent a certain social group, for example, is obviously linked to motives of return on investment. Since algorithmic processing of information is a business first and foremost (and an extremely lucrative one), algorithm owners are spurred by financial incentives to favor social groups that are best positioned to act upon the information received by exchanging their capital for services. Again, and as above, this paper isn’t making a judgment call on whether this is right or wrong, or better or worse than the previous system of human gatekeepers. Journalism, in fact, has always been a business (more or less incentivized by the state), and media outlets have always needed to balance the desire to inform without economic pressures on the choice of news, against that same economic pressure of making payroll. The main difference with the new system is the lack of ethical and moral codes governing or influencing algorithms. In fact, partly due to the common perception of neutrality of algorithms, and partly because of the great speed with which algorithms made themselves irreplaceable (and inevitable), the public sphere still needs to catch up on codifying and defining moral and ethical policies, or frameworks, that would govern or influence algorithm owners.

Currently, in fact, not only there is no commonly accepted mechanism to oversee the creation and functioning of algorithms, but algorithm owners themselves are effectively keeping their algorithms secret. Algorithm owners take legal action to protect their technology as trade secrets, or vital components of a business that provide their edge on the market, and as such are legally protected from prying eyes. This is the phenomenon of black box algorithms - the most common automated information parsing processes, such as the Google search engine, or the Facebook wall, are completely locked and cannot even be analysed by academics to understand whether there is bias at play⁶. The corollary to this is if external groups cannot evaluate algorithms, there is no way that independent groups that have been designed to minimize implicit cognitive bias can evaluate them, which means that, short of profound legislative reform, the information that society receives will effectively stay locked into the cognitive bias of algorithm owners.

There is another implicit bias at play with algorithms: a user-defined one. Algorithms are usually built so that they self-refine the correctness of their results based on user input - by statistically analysing data like which results are clicked on most frequently, which web pages are read the longest, tracking mouse moving patterns on a page, etc. This functionality is generally welcome, since it helps algorithmic processing of information to combat temporal staticity and react to changes in user behaviour. However, if an algorithm self-updates its processes solely upon the quality of user input, it will learn and start to incorporate implicit biases of the majority group - with the risk of automatically excluding less-represented social groups and lock itself in a self-perpetuating relevance confirmation loop (see examples below).

Bias in algorithmic systems can exacerbate inequality

One prime example of developer bias perpetuating inequality is in gender selection forms. Gender identity is a complex and nuanced topic, and is arguably one of the fundamental aspects of individual definition of identity. Therefore, when requesting information about an individual on an online platform — in order to, for example, provide a service or better tailor results to that specific individual — online interfaces should be able to accommodate all the possibilities of human interaction with regards to assigning self-identity. The choices around design of online forms, however, are usually delegated to developers — people with specific technical deep skills that don’t commonly comprise gender studies and, as we saw before, bring with them a very specific and commonly shared cultural background and accompanying implicit biases. This usually results in online gender selectors representing solely the binary male-female option, that inevitably marginalizes any person that doesn’t identify with one of these two options. This becomes an even larger problem in cases where users are required to match their legal gender at birth, because it also marginalizes those people that define exclusively with one gender, that however isn’t the gender they were born with. There are some examples of proactively trying to accommodate multiple gender identities: in 2014, Facebook compiled a list of 58 gender options for users to choose from (Oremus, 2014). The Indian government also officially expanded their gender definition to a “third sex” option (Nelson, 2014).

Explicit bias from algorithm owners can also exacerbate inequality. Facebook, for example, racially profiles users based on their browsing habits and serves those different groups different ad results based on their algorithmically computed race (Newitz, 2016). This racial profiling (or, as Facebook calls it, “ethnic affinity”) became obvious when, for the promotion of the 2016 film “Straight Outta Compton,” Facebook served three different trailers respectively for white, African-American and Hispanic audiences. The content of the trailers was also radically different and appealed to completely different traits of the film — and the decision on which aspects of the movie to show to separate audiences had obvious roots in implicit bias and prejudice. As Annalee Nevitz from Ars Technica puts it when analysing differences between the “white ethnic affinity” and “African American ethnic affinity” trailers:

The two trailers aren't just mildly different—they look like they're advertising two completely different films. The version for white users [...] comes across like a gangster movie. It emphasizes the violence of the group, showing them brandishing semi-automatics, clashing with police, and walking through what appear to be riots. We only see the actors without seeing any of the actual members of N.W.A. who appear in the film. It looks like a scripted drama and not a biography of real people.

But the version of the trailer for black users [...] plainly depicts the film as a biography. We see members of N.W.A. talking about how their music was "protest art" that reflected the horrible conditions in LA ghettos of the 1980s. A quiet ripple of music swells behind them as they talk to people about what N.W.A. meant to them, and the movie comes across as a reverent retelling of how these important African American artists rose to fame. We don't see any violence until over a minute into the trailer, after it has been contextualized as "protest" imagery created by the very relatable, beloved members of N.W.A. (Newitz, 2016, para. 3)

When speaking of self-evolution of algorithms and the risk of majority’s implicit or explicit bias to influence how algorithms automatically fine-tune their results, an apt example is Latanya Sweeney’s 2013 research into racial discrimination of ads served through the Google Ads platform. The study shows that:

First names, previously identified by others as being assigned at birth to more black or white babies, are found predictive of race (88% black, 96% white), and those assigned primarily to black babies, such as DeShawn, Darnell and Jermaine, generated ads suggestive of an arrest in 81 to 86 percent of name searches on one website and 92 to 95 percent on the other, while those assigned at birth primarily to whites, such as Geoffrey, Jill and Emma, generated more neutral copy: the word "arrest" appeared in 23 to 29 percent of name searches on one site and 0 to 60 percent on the other. (Sweeney, 2013 abstract)

What is particularly important in this example is that Sweeney’s research suggests there is no human element behind the choice to racially discriminate, either on behalf of the client, or the advertiser. Rather, the research into the mechanics of how Google Ads visualize specific keywords and copy show that they are based on the automated learning method of the algorithmic process. In the specific case in question, the client delivered a set of different copy examples that algorithms randomly connected to names when showing ads, not preferring one over the others. It is arguably the human interaction with those links that spurred racial discrimination: algorithms assign different weights to different versions of the same ad based on the number of clicks a certain version of ads receives. In fact, it is a financially sound decision for both Google Ads and the client: the client doesn’t get charged each time an ad is visualized, they get charged once a user clicks on an ad. It is thus in both parties’ interest that those ads that get clicked the most get visualized the most within the respective parameters. The fact that names assigned primarily to black people garnered more clicks than those assigned to white people is indicative of racial discrimination in users — but algorithms learn from this interaction in a completely autonomous way, and with time start to replicate and amplify the same discriminations. It is an open question whether Google has a role in curbing this phenomenon or not, and again, the “right or wrong” quality here is a moot point for the purposes of this paper. However, before interactive ads, there was no way to automatically tailor advertising or, if we broaden the scope, any presented information, based on user interaction performed by the majority, and in real time.

Explicit bias towards positive information limits exposure

Let’s turn our attention back to algorithm owners. The Facebook wall, i.e. the stream of news and updates on a user’s social connections, is algorithmically managed. To mitigate the signal to noise ratio and provide to users a subset of content that is cognitively manageable, Facebook uses algorithmic prioritization and assignment of weights to posts. One defining factor for automated parsing is positivity: posts that express gratitude, celebration, and positive sentiment in general hold more weight than posts that express sadness, rage, or negative sentiment. This phenomenon became obvious in 2014 after the incident in Ferguson, Missouri in the united States. Mike Brown, a black youth, was shot and killed by a white police officer seemingly without probable cause. Videos and eyewitness testimony show that Brown was kneeling, with his hands behind his head and his back turned to the police officer, when the police officer fired multiple shots, ending Brown’s life. This event sparked outrage in the black community and spread virally over the Internet on a number of social media — but not on Facebook. In fact, as Tufekci recounts,

Acting through computational agency, Facebook’s algorithm had “decided” that such stories did not meet its criteria for “relevance”—an opaque, proprietary formula that changes every week, and which can cause huge shifts in news traffic, making or breaking the success and promulgation of particular stories or even affecting whole media outlets. By contrast, Twitter’s algorithmically unfiltered feed allowed the emergence of millions of tweets from concerned citizens, which then brought the spotlight of the national media. Algorithmic filtering also by Twitter might have meant that a conversation about police accountability and race relations that has since shaken the country might never have made it out of Ferguson. (Tufekci, 2015, p. 213)

Twitter was the social media platform that let Ferguson reach critical mass and galvanize the public sphere. At the time, Twitter didn’t algorithmically manipulate its feed — all tweets from a user’s network were shown chronologically. Twitter was how news journalists got wind of the event and provided the amplifying effect that helped the #Ferguson phenomenon grow. If there was no Twitter, or if all social media news sharing platforms filtered their news algorithmically, there is a real possibility that Ferguson wouldn’t have turned into a highly important newsworthy event that spurred a new political activism movement, #BlackLivesMatter, highly active to this day across the United States in advocating for race equality.

From a financial standpoint, it makes sense for Facebook and similar platforms to filter for sentiment — we can safely assume that algorithm owners, if they run a for-profit venture, tweak the algorithms to maximize their profits. An obvious way of doing that is prioritising the types of posts that users on average are more willing to interact with, since higher levels of interaction mean more time spent on the platform, and an increased chance of viewing and interacting with advertising. This phenomenon does however suggest that algorithm owners fully control the chain of decisions over which information the public is going to be privy to, and this chain of decisions has its roots (and its ultimate goal) in securing financial profit.

⁶. However, researchers are doing their best to reverse-engineer algorithmic parsing choices: in May 2016, Carnegie Mellon researchers published a paper on a new system for detecting bias in otherwise opaque algorithms through quantitative analysis of results based on well-defined inputs. (Datta, Sen, & Zick, 2016) ↩