MaryAnn's Blog

I have no blog post for these two weeks on account of winter break from school. Happy holidays, MaryAnn

This week was finals in school, so I only had one brief meeting with my mentor and Caitlin. Caitlin and I presented our idea about mapping the reliability curves of protected groups to a compromised line and adjusting the model accordingly. Caitlin had spent time considering the same problem but in the context of equalized odds. We found in the meeting that we were saying many of the same things, and that we were unclear as to the distinction between the two (equalized odds and calibration). Here's what we found: Equalized odds and calibration both deal with minimizing the discrepancy of error in prediction. (note that they are not  about minimizing the error). What does that mean? Consider the following three scenarios (I created these contrived examples in Microsoft Excel for purpose of discussion): Figure 1 Figure 2 Figure 3 For each of these scenarios, one could make the argument that the discrepancy of error is the same. In Figure 1, the error in one is underes...

In the previous post, I talked about a reliability diagram to quantify error in a calibration model. Caitlin and I spent time this week discussing the next step: correcting for an unfair model. Disclaimer: I created the plots in this post with Microsoft Excel, and they don't reflect any real data. What does unfairness look like? For a running example, let's imagine we were evaluating a model that attempts to predict one's salary. This model will be used to calculate future salaries for new hires, so it is important that the model is not biased against any protected attributes, for example, sex. We have a large dataset that includes salary information for roughly the same number of males and females. We run the model with this labeled dataset and get the following reliability diagram for salary: In this plot, points on the diagonal black line represent a perfect prediction. Points above the line represent overestimation (true salary was lower than we expected), and...

At the end of the last post, I mentioned that we would be looking at the Audit task for Calibration and Equalized Odds under the Scoring application. Caitlin and I divided these tasks; I started looking at error metrics for Calibration and thinking about how we could apply it to scoring. Caitlin did the same for Equalized Odds. Calibration in Classification In my previous post, I described calibration under classification. In this paradigm, one is trying to predict whether or not an individual is in the positive class. Each individual is assigned a probability; the probability that the individual is in the positive class. If a classifier is well-calibrated, approximately x% of people assigned a probability of x% will be positive. Basically, calibration means that the probability estimate means what is says it means. Problems of applying directly to Scoring Scoring can be thought of as linear regression. In this paradigm, individuals are assigned a score which represents their v...

This week, I met with the graduate student, Caitlin, to discuss next steps. (Unfortunately, I missed the weekly meeting with the professor because I was at an interview.) We decided on a strategy for addressing each aspect of ranking and fairness. This strategy is what we are calling 3x3x2: 3 fairness criteria, 3 applications, and 2 tasks. 3 Fairness Criteria In my last post, I summarized a series of papers related to fairness criteria for classification. After my summary, I introduced three distinct ideas: statistical parity, calibration, and equalized odds. Statistical Parity Statistical parity is the idea that two groups should have equal outcomes. For example, if you assume that women are just as qualified to attend graduate school as men, then an equal number of women should be admitted to graduate programs as men. This criteria was described by Friedler et. al as the "We're All Equal" (WAE) axiom. In short, the WAE axiom says that with respect to the decision...