Week 12: Error metrics for calibration and equalized odds

At the end of the last post, I mentioned that we would be looking at the Audit task for Calibration and Equalized Odds under the Scoring application. Caitlin and I divided these tasks; I started looking at error metrics for Calibration and thinking about how we could apply it to scoring. Caitlin did the same for Equalized Odds.


Calibration in Classification

In my previous post, I described calibration under classification. In this paradigm, one is trying to predict whether or not an individual is in the positive class. Each individual is assigned a probability; the probability that the individual is in the positive class. If a classifier is well-calibrated, approximately x% of people assigned a probability of x% will be positive. Basically, calibration means that the probability estimate means what is says it means.

Problems of applying directly to Scoring

Scoring can be thought of as linear regression. In this paradigm, individuals are assigned a score which represents their value in the ranking (i.e. a high score is better than a low score). In a supervised learning area, we would know the true score for each individual. This is not really the same problem as classification. We are not looking for a probability estimate of whether someone is in the positive class, because there is no positive class.

How I addressed it

The first thing I considered doing was trying to force each individual to have a probability value. This could be done by assuming the true scores follow some kind of underlying distribution and trying to learn a p-value that represents the likelihood that the score we assigned the individual was really the true score. This doesn't make much sense for two reasons: one, we are assuming that individuals are individual, and we can't learn a p-value on just one data point. Second, we won't be able to determine if the probability is well-calibrated. My guess is that a well-calibrated version of this model means that all p-values are the same.

I decided to turn to Google and look for alternative definitions of calibration that could make more sense for regression. One of the top hits was this post on StatsExchange. The poster, Berk Ustun, was looking for an alternative method for testing performance. He later resolved his own post and cited a couple of papers that describe error metrics, specifically calibration.

Root Mean Squared Error (RMSE) and Reliability Diagram

In this survey paper mentioned in the StatsExchange resolution post, the authors mention a calibration metric that is based on reliability diagrams.

A reliability diagram compares the predicted value to the proportion of true positives. In classification, the X axis could be thought of as the probabilistic prediction for each individual, ordered from lowest to highest. In a perfect prediction (perfectly calibrated), the plot would be a straight line. In this image, the blue line represents the reliability of their predictor.

Source: https://www.mathworks.com/matlabcentral/mlc-downloads/downloads/submissions/25704/versions/4/screenshot.png
In this example, for individuals for whom the predictor assigned a probabilistic score of 0.7 (70% likely to be in the positive class), about 80% were positive. This means the model under-predicted these individuals. For individuals assigned a probabilistic score of 0.4 (40% likely to be in positive class), about 40% were truly positive. The model was well calibrated at this threshold.

I believe this same model can work in a regression setting. On the X axis, I would have the scores of individuals ordered from lowest to highest. On the Y axis, I would have the true scores (supervised learning). I could fit a line of these points that would look very similar to the one above.

In the paper, the authors suggest using a Mean Calibration Error to evaluate the accuracy of a predictor. This metric is based off of root mean squared error (RMSE). I think this same metric can be applied directly to calibration.

I discussed this with Caitlin and with our mentor, and it seems to be like a good way to go. The next step is to determine how to fix a model that is not well calibrated. Until next week!

Comments

Post a Comment

Popular posts from this blog

Week 31: Datasets for fairness

This week, Diana and I looked for some datasets on which to evaluate our fairness criteria. This is no simple task. For one, the datasets need to be large enough to be split into training, calibration, and testing sets. Second, there needs to be some protected group attribute like sex, race, or age. Third, there needs to be some outcome ranking as well as some truth value. The truth is often the hardest attribute to find. In this post, I introduce some of the datsets we decided to use. I try to identify each of the necessary features that we need to properly test our fairness criteria. COMPAS The COMPAS dataset is an obvious choice because it is well ingrained in the literature about fairness. It has 18,000 observations (n=18000) and 1000 attributes (k=1000). Protected groups can be sex (81/19) or race (51/49). The outcome attribute can be the recidivism score, the score used in court that represents the risk that the accused will commit another crime. The truth attribute is a li...
Read more

Week 32-33: Wrapping up

Hi all. This is my last blog post for this project. The last few weeks, we spent little time working on the project and instead focused on finishing up our classes. To be honest, we lost some motivation after finishing the paper, and we've been reluctant to start up new tasks so close to the end of the year. In this post, I'd like to talk about some of the things I've learned in this project other than what we found in our research. This post is really more a service to myself; to help me reflect on this past year and what I've learned. When I started this project, I didn't understand how research in computing worked. My exposure to research had been more biology/clinical, in which research involved setting a hypothesis about some protein, designing an experiment with positive and negative controls, and then running the experiment repeatedly until confident in the results. In computing research, my experience has been of two varieties: reading papers and proce...
Read more

Week 30: Returning to Fairness

With the LineUp paper submitted, it's now time to return to the fairness problem we were working on before. As a bit of a refresher for you and for me, I'll review what we've done with fairness so far and what remains to be done. Fairness in Ranking Up until now, the literature regarding fairness has focused on fairness in classification. As an illustrative example, consider fairness in hiring practices with repect to applicant sex. A job opening might receive 100 applicants evenly split into 50 males and 50 females. If males and females are equally capable of doing the job, then you might be concerned to see the hiring company mostly interviewing male applicants. To evaluate fairness, you would compare each applicant's capability (capable of performing duties, not capable) to their outcome (interviewed, not interviewed). In a completely fair scenario, the same proportion of capable applicants from each sex group would receive an interview. Small deviances from this...
Read more