This repo contains code that evaluates racial and gender bias of two NLP cloud services: Amazon's AWS Comprehend and Google's Google Cloud Natural Language.
Both show statistically significant racial and gender bias in sentiment rating on a test corpus described here.
The EvaluateSentimentScores.ipynb notebook loads the corpus and runs sentences from the corpus on the Google and AWS services respectively. The notebook EvaluateCloudSentimentBias.ipynb then computes statistical significance of the results and produces a set of visualizations.