Week 23: Football!

Last week, I was able to write a script that learns a linear model and performs an isotonic regression to redistribute the outcomes. We ran this model using the College Scorecard data paired with US News college rankings. Unfortunately, the number of colleges available in our data was so small that we were unable to split our data into training, calibration, and testing and still have enough to fill 10 bins. So this week, my goal was to find a new dataset with lots of data.

I decided to look at football rankings. During football season the NCAA posts weekly rankings of who they think the best teams are going to be (see). From the first ranking (pre-season) to the last (post-season), these rankings change drastically. My idea was that fairness in this context could be something to do with the college, such as public or private school or number of years the football program has been around (binarized), and that NCAA rankers could be biased against one type or the other. The problem is a bit contrived, but the goal was really to get our hands on bigger data.

So I scraped the ESPN website for pre-season and post-season rankings in 2016 and 2017 and joined it with statistics about the age of the football program and whether the college was public or private. Skipping the linear model formation step, I used the 2016 pre- and post-rankings to learn an isotonic regression split by public/private. I then transformed the 2017 pre-ranking with this regression and compared it to the post-ranking.

There wasn't much to see. I think it is because the dataset that I chose was actually kind of horrible for this. The NCAA page only ranks the top 25 colleges, and there are 124 colleges in the FCS. Not only this, the universities in the top 25 changed from week to week, so the only colleges for which we had all four data points were, like, Alabama.

This was about the extent of what I covered this week. I want to try this again, perhaps with a better dataset. Here's a link to one that was used by PODIUM, a project at Georgia Tech that's a lot like ours.

Comments

Post a Comment

Popular posts from this blog

Week 31: Datasets for fairness

This week, Diana and I looked for some datasets on which to evaluate our fairness criteria. This is no simple task. For one, the datasets need to be large enough to be split into training, calibration, and testing sets. Second, there needs to be some protected group attribute like sex, race, or age. Third, there needs to be some outcome ranking as well as some truth value. The truth is often the hardest attribute to find. In this post, I introduce some of the datsets we decided to use. I try to identify each of the necessary features that we need to properly test our fairness criteria. COMPAS The COMPAS dataset is an obvious choice because it is well ingrained in the literature about fairness. It has 18,000 observations (n=18000) and 1000 attributes (k=1000). Protected groups can be sex (81/19) or race (51/49). The outcome attribute can be the recidivism score, the score used in court that represents the risk that the accused will commit another crime. The truth attribute is a li...
Read more

Week 32-33: Wrapping up

Hi all. This is my last blog post for this project. The last few weeks, we spent little time working on the project and instead focused on finishing up our classes. To be honest, we lost some motivation after finishing the paper, and we've been reluctant to start up new tasks so close to the end of the year. In this post, I'd like to talk about some of the things I've learned in this project other than what we found in our research. This post is really more a service to myself; to help me reflect on this past year and what I've learned. When I started this project, I didn't understand how research in computing worked. My exposure to research had been more biology/clinical, in which research involved setting a hypothesis about some protein, designing an experiment with positive and negative controls, and then running the experiment repeatedly until confident in the results. In computing research, my experience has been of two varieties: reading papers and proce...
Read more

Week 30: Returning to Fairness

With the LineUp paper submitted, it's now time to return to the fairness problem we were working on before. As a bit of a refresher for you and for me, I'll review what we've done with fairness so far and what remains to be done. Fairness in Ranking Up until now, the literature regarding fairness has focused on fairness in classification. As an illustrative example, consider fairness in hiring practices with repect to applicant sex. A job opening might receive 100 applicants evenly split into 50 males and 50 females. If males and females are equally capable of doing the job, then you might be concerned to see the hiring company mostly interviewing male applicants. To evaluate fairness, you would compare each applicant's capability (capable of performing duties, not capable) to their outcome (interviewed, not interviewed). In a completely fair scenario, the same proportion of capable applicants from each sex group would receive an interview. Small deviances from this...
Read more