We Graded Our Own Week 1 Rankings. Here's How We Did.

Every Thursday we publish weekly rankings. Every Tuesday, starting now, we're going to tell you how they did.

Not with a vague "we nailed some, missed some." With actual numbers, graded the same fixed way every week, and posted whether the week was good or bad. This is week one of that, and we're going to be upfront about something before we start: one week of results tells you almost nothing. The whole point of doing this every week is to figure out, over a season, which parts of our rankings are real signal and which parts are noise. Week 1 is the first data point, not the verdict.

Here's Week 1.

Now the longer version.

How we're grading this

Rankings are a list of names in order. The question is whether the order was right. We look at that two ways, from simplest to most useful:

01Did our starters finish as starters?

Running back was the reliable position, as it usually is — just over 70% of the guys we called starters were starters. Quarterback was a coin flip: Caleb Williams was the QB1 of the week and we had him 10th. Bryce Young was QB3 and we had him 23rd. Kyler Murray, our QB14, scored less than a point.

Tight end was rough, and the reason has a name. Colston Loveland was our TE1. He played, saw two targets, caught neither, and scored 0.0. That isn't a "didn't play" asterisk — he was active, and he counts as a zero for anyone who started him. One player at the top of a position finishing dead last is the single biggest ranking mistake we made in Week 1, and we'll be watching whether that's a Week 1 thing or a Loveland thing.

02The hard calls

Two numbers per position here. All calls is every pair in the bubble, counted equally. Close calls is the same pairs, but weighted so the ones ranked right next to each other count the most — because RB24-vs-RB25 is the decision you actually lose sleep over, and RB19-vs-RB30 isn't a decision at all.

The shaded range is the noise band: with only twelve players in the bubble, the "true" number could plausibly be anywhere in there. When the band includes 50%, we can't honestly claim the rankings beat a coin flip that week.

Read that table honestly and here's what it says: we were above 50% at every position on the easy calls, and above 50% at three of four on the close calls — but every single band includes 50%. We can't yet prove any of it is skill. RB and WR look the most promising; TE looks like we had no idea. But "looks like" is doing a lot of work in that sentence, and that's the point.

Our best and worst bubble calls of the week:

What we're taking from Week 1

• Tight end is the position to watch. Four of our twelve TE starters finished as starters, our TE1 scored zero, and the close-call number was the only one under 50%. If there's a part of the rankings that needs work, it's this one — but we want three more weeks before we touch the model.
• We can't separate signal from noise yet, and we're not going to pretend we can. Every bubble number is inside its noise band. By Week 4 or 5 those bands will be tight enough to say something real. Until then, the numbers are a record, not a conclusion.
• QB was noise. The QB1 of the week was our QB10. The QB3 was our QB23. Our QB14 scored under a point. Some weeks are like that, and we'd rather find out over ten weeks whether our quarterback rankings are any good than decide after one.

We'll add Week 2 to this next Tuesday, and we'll start showing season-to-date numbers so one weird week — good or bad — stops being the whole story.