Ranking Jeff Sagarin

As any fan of college football knows, Jeff Sagarin has developed a mathematical model for ranking college football teams, which is published in USA Today. His rankings are more than just an academic exercise, because they are one of the six computer models used in the BCS (in other words, real dollars are at stake).

Sagarin, like most of the computer rankers, keeps his actual methodology secret. (The Colley Matrix, also used in the BCS, is a welcome exception, as Wesley Colley gives makes his formula publicly available.)

I have long been suspicious of the methodology, and utility, of the Sagarin rankings. Often his rankings seem to defy logic. From what Sagarin is willing to reveal, his rankings take into account strength of schedule and margin of victory, among other factors. The BCS requested Sagarin create a new formula that does not take victory margin into account for its purposes, and Sagarin calls this method the “ELO_CHESS”. The BCS friendly “ELO_CHESS” ranks the top 5 as follows-

1. Ohio State
2. Michigan
3. Florida
4. USC
5. LSU

Sagarin however feels that this BCS requested “ELO_CHESS” is “less accurate” than his regular methodology. In his regular formula, his current rankings are –

1. Ohio State
2. USC
3. Michigan
4. LSU
5. Louisville
6. Florida
7. California
8. West Virginia
9. Boise State
10. Tennessee

I’m not going to waste print space analyzing this top ten, other than to point out the mysteries of LSU being 2 spots above a team it lost to by 13 points, or that a USC team that lost two times to unranked teams is listed second.

Or, think of it this way - if Jeff Sagarin was picking the BCS title game, it would be Ohio State against USC.

Sagarin states that his method is accurate for making “predictions” for games (he even calls it the “Predictor”). In the predictor, you take the number assigned to the team – like the 98.25 rating for Ohio State, and compare it to another team, to see what the winning margin is predicted to be. OSU’s 98.25 to Florida’s 87.02 “predicts” that the Buckeye’s will win 11 points.

Well, let us put Mr. Sagarin to the test this bowl season.

For the games played so far, here are his rankings, the predictions and the results.

33rd TCU v 84th Northern Illinois – TCU by 15
Actual – TCU by 30. Missed by 15 points.

21st BYU v 22nd Oregon – BYU by 5
Actual – BYU by 30. Missed by 25 points.

64th Rice v 103rd Troy – Rice by 9
Actual – Troy by 24. Missed by 31 points.

85th New Mexico v 69th San Jose State – New Mexico by 1
Actual – SJSU by 8. Missed by 9 points.

58th Utah v 55th Tulsa – Utah by 2
Actual – Utah by 12. Missed by 10 points.

28th Hawaii v 27th Arizona State – Hawaii by 4
Actual – Hawaii by 17. Missed by 13 points.

So, thus far, Sagarin has picked the winners in 4 of 6, but in each case, he greatly underestimated the margin of victory. His average point miss through 6 games is 17.17.

If you were playing “Vegas” by his numbers, you would be broke.

If you look at Sagarin’s rankings, he ranks Pac Ten teams Oregon and ASU right near their bowl opponents, yet those opponents far exceeded the expectations of his “Predictor”. Sagarin also ranks conferences. Presently, the Pac Ten is his top rated conference, with the SEC second. In his strength of schedule rankings, Pac 10 Teams have by far the “hardest” schedules, with all ten top rankings going to the conference. His SOS rankings –

1. Stanford
2. USC
3. UCLA
4. Arizona
5. Washington
6. California
7. Oregon
8. Washington State
9. Arizona State
10. Oregon State

It defies credibility to believe that every Pac Ten team played a tougher schedule this year than anyone else in the nation.

It isn’t hard to see here that Sagarin’s methodology has some fundamental flaws that severely impact his rankings. Since his formula is propriety, we can only speculate to what that is, but I suspect the following –

- An “echo” effect. When a team like USC plays tough out of conference opponents like Arkansas, Nebraska and Notre Dame, their strength of schedule gets attributed to every team they play. Since the same conference teams play each other, it "echoes" throughout the conference.

- An overemphasis on high scoring teams. Sagarin admits he takes scoring margins into account. USC, for instance, was averaging over 32 points a game before they ran into UCLA. UCLA also happens to be the best defensive scoring team USC faced all year, giving up 17.9 points per game.

Several teams also played Notre Dame (Stanford, UCLA and USC), so ND’s relatively good record also helped the conference)

Sagarin’s method does not seem to take into account the effectiveness of good defenses. That best defense of UCLA that USC played (and lost to)? Well, it is ranked 27th nationally. And that 27th ranked defense made it very evident that USC could be shut down.

The problem is this - just because some teams in a conference play some quality out-of-conference opponents does not mean that the entire conference itself is good. USC, to their credit, played and beat Arkansas, Nebraska and Notre Dame. USC is a pretty good team.

However, there is really no evidence the rest of the Pac Ten is very good. Cal, for instance, played 3 out-of conference opponents - Tennessee, Minnesota and Portland State. They got whipped by UT, beat 6-6 Minnesota, and beat Division I-AA Portland State. Cal had a pretty mediocre out of conference record. Yet, they are second best team in the Pac Ten.

Whatever Sagarin is doing, it has a real effect on his ratings. USC, with two losses, is ranked 2nd. California, with 3 losses, is raked 7th, 3 places ahead of 3 loss Tennessee which destroyed Cal head-to-head. Stanford – 1 win Stanford – is ranked 91st, ahead of teams with 5, 6 or even 7 wins.

We are going to keep watching Sagarin here through the bowl season.

And, in case you want to watch also, his predictions for the next 2 bowl games have Central Michigan as 4 points better than Middle Tennessee, and UCLA as 5 points better than Florida State.