Welcome! I intend this to be an ongoing project of predicting NFL game outcomes, point spreads, and final scores. My hope is that my models will improve over time as they become more sophisticated and use better data. I will try to regularly publish predictions to keep myself accountable, and will make observations along the way about what is working and what isn't. See below for latest updates. Enjoy!
Week 14: Fifth Published Predictions (Win/Lose)
Previous predictions:
_________________________________________________________________________________
Week 14 Recap:
Well, although I was confident of predicting 10 or 11 games correctly, my predictions went 9-7. I suppose that will happen every now and then with so many close games. How likely is that to happen normally? I ran a simulation to find out.
I took the probability of winning for each of the 16 games in week 14 (only using those predicted to win), and used a random uniform distribution function to assign another probability outcome to that game. If the original probability was higher than the probability outcome, then a win was assigned, otherwise, a loss. For example, suppose a team was predicted to win at 0.70. Then if the probability outcome was anywhere between 0 and 0.70, that team "won" and the prediction was correct. If it was from 0.71 to 1.0, the team lost and the prediction was incorrect. The total number of correctly predicted wins for the 16 games was summed up (e.g., 9 correctly predicted wins of the 16 games) and recorded. I repeated the process 100,000 times with all results recorded.
The results were approximately normally distributed, ranging from 2 correct predictions to 16 correct predictions. The most common outcomes were: 10 (20%), 9 (19%), and 11 (17%) correct predictions. So more than 50% of the time, my predictions (with similar probabilities) should fall between 9 and 11 correct predictions. About 80% of the time my predictions should fall between 8 and 12 correct. This gives more substance to my use of confidence below, which upon reflect, was a bit overly confident.
I took the probability of winning for each of the 16 games in week 14 (only using those predicted to win), and used a random uniform distribution function to assign another probability outcome to that game. If the original probability was higher than the probability outcome, then a win was assigned, otherwise, a loss. For example, suppose a team was predicted to win at 0.70. Then if the probability outcome was anywhere between 0 and 0.70, that team "won" and the prediction was correct. If it was from 0.71 to 1.0, the team lost and the prediction was incorrect. The total number of correctly predicted wins for the 16 games was summed up (e.g., 9 correctly predicted wins of the 16 games) and recorded. I repeated the process 100,000 times with all results recorded.
The results were approximately normally distributed, ranging from 2 correct predictions to 16 correct predictions. The most common outcomes were: 10 (20%), 9 (19%), and 11 (17%) correct predictions. So more than 50% of the time, my predictions (with similar probabilities) should fall between 9 and 11 correct predictions. About 80% of the time my predictions should fall between 8 and 12 correct. This gives more substance to my use of confidence below, which upon reflect, was a bit overly confident.
Big upsets (that I got wrong): Lions/Rams, Bears/Redskins, Steelers/Bengals, Giants/Dolphins
Big surprises (that I got correct): Broncos/Raiders
_________________________________________________________________________________
Week 14: Fifth Published Predictions (Win/Lose)
My code is finally automated enough that I can regularly make predictions. This week I focused on making sure my models generalized well (i.e., separate training, testing, and evaluation data for comparison) and were stable. Why does this matter? If you don't care, you can skip a few paragraphs. If you do, read on.
Suppose that you have a model (like mine) that predicts with about 80% accuracy on the data used to create the model, but only predicts with about 64% accuracy on data it hasn't seen. Since we care about predicting data we have not seen, what matters most is the accuracy of a model with regard to new data. So the model should perform equally well with training data that it has seen, and test data it hasn't seen. That way, we can be confident that it will perform well with predictions that even we haven't seen.
Also, if the model accuracy is jumping around a lot even when using the same columns of data because your random sampling of data used in training and testing significantly impacts the stability of the model, that is also bad. You want to be confident that your model accuracy will be, say, between 65% and 67% each time and not, somewhere between 60% and 70%. You may get a lucky prediction, but you don't want to rely on luck. You want to be confident that your predictions will be as consistently accurate as your model claims they will be.
Having focused on generalizability and stability, the best model for predicting wins generalized to about 66% (training and testing) and stabilized to between 65% to 67%. I used a logistic model for this.
If you skipped some paragraphs, you can now rejoin. What does the above mean practically? It means that since there are 16 games being played this week, I am confident that I will get 10 or 11 game predictions correct. No more, no less.
I also added predicted score differences and predicted final scores. But don't use them for betting yet. Score differences have an average error of 13 and an r^2 of 0.23. Final scores are off on average by 10 points with an r^2 of 0.17. Since almost all predicted point differences are smaller than 13 and almost all scores are within 10 of each other, they are analytically useless. So only use them for your enjoyment.
Notice also that the predictions do not agree with each other in many cases. For example, the 49ers play the Browns, and are predicted to win with 55% probability. However, they have a -1 point difference (so they should lose by 1), and the final score is predicted to be 20-20 (so they should tie). I did round to integers and that can help explain a little, but the primary issue is that these are different models and will give slightly different results, especially since these two new models aren't very good yet. In short, predicting wins and losses is much easier than predicting scores and point differences. But I'll keep at it.
Suppose that you have a model (like mine) that predicts with about 80% accuracy on the data used to create the model, but only predicts with about 64% accuracy on data it hasn't seen. Since we care about predicting data we have not seen, what matters most is the accuracy of a model with regard to new data. So the model should perform equally well with training data that it has seen, and test data it hasn't seen. That way, we can be confident that it will perform well with predictions that even we haven't seen.
Also, if the model accuracy is jumping around a lot even when using the same columns of data because your random sampling of data used in training and testing significantly impacts the stability of the model, that is also bad. You want to be confident that your model accuracy will be, say, between 65% and 67% each time and not, somewhere between 60% and 70%. You may get a lucky prediction, but you don't want to rely on luck. You want to be confident that your predictions will be as consistently accurate as your model claims they will be.
Having focused on generalizability and stability, the best model for predicting wins generalized to about 66% (training and testing) and stabilized to between 65% to 67%. I used a logistic model for this.
If you skipped some paragraphs, you can now rejoin. What does the above mean practically? It means that since there are 16 games being played this week, I am confident that I will get 10 or 11 game predictions correct. No more, no less.
I also added predicted score differences and predicted final scores. But don't use them for betting yet. Score differences have an average error of 13 and an r^2 of 0.23. Final scores are off on average by 10 points with an r^2 of 0.17. Since almost all predicted point differences are smaller than 13 and almost all scores are within 10 of each other, they are analytically useless. So only use them for your enjoyment.
Notice also that the predictions do not agree with each other in many cases. For example, the 49ers play the Browns, and are predicted to win with 55% probability. However, they have a -1 point difference (so they should lose by 1), and the final score is predicted to be 20-20 (so they should tie). I did round to integers and that can help explain a little, but the primary issue is that these are different models and will give slightly different results, especially since these two new models aren't very good yet. In short, predicting wins and losses is much easier than predicting scores and point differences. But I'll keep at it.
Here are the predictions for Week 14:
| Week |
Date
|
Team
|
Home Away
|
Opponent
|
Prob. Win
|
Pred. Team Win
|
Pred. Team PD
|
Pred. Team Score
|
Pred. Opp. Score
|
Actual Team Win
|
Actual Team PD
|
Actual Team Score
|
Actual Opp. Score
|
14
|
12/10/2015
|
Cardinals
|
Vikings
|
0.572
|
1
|
2
|
26
|
23
|
1
|
3
|
23
|
20
| |
14
|
12/10/2015
|
Vikings
|
@
|
Cardinals
|
0.428
|
0
|
-2
|
23
|
26
|
0
|
-3
|
20
|
23
|
14
|
12/13/2015
|
49ers
|
@
|
Browns
|
0.552
|
1
|
-1
|
20
|
20
|
0
|
-14
|
10
|
24
|
14
|
12/13/2015
|
Bears
|
Redskins
|
0.699
|
1
|
3
|
28
|
24
|
0
|
-3
|
21
|
24
| |
14
|
12/13/2015
|
Bengals
|
Steelers
|
0.596
|
1
|
2
|
26
|
24
|
0
|
-13
|
20
|
33
| |
14
|
12/13/2015
|
Bills
|
@
|
Eagles
|
0.577
|
1
|
2
|
25
|
23
|
0
|
-3
|
20
|
23
|
14
|
12/13/2015
|
Broncos
|
Raiders
|
0.479
|
0
|
1
|
22
|
21
|
0
|
-3
|
12
|
15
| |
14
|
12/13/2015
|
Browns
|
49ers
|
0.448
|
0
|
1
|
20
|
20
|
1
|
14
|
24
|
10
| |
14
|
12/13/2015
|
Buccaneers
|
Saints
|
0.482
|
0
|
1
|
26
|
27
|
0
|
-7
|
17
|
24
| |
14
|
12/13/2015
|
Chargers
|
@
|
Chiefs
|
0.424
|
0
|
-4
|
21
|
26
|
0
|
-7
|
3
|
10
|
14
|
12/13/2015
|
Chiefs
|
Chargers
|
0.576
|
1
|
4
|
26
|
21
|
1
|
7
|
10
|
3
| |
14
|
12/13/2015
|
Colts
|
@
|
Jaguars
|
0.618
|
1
|
3
|
25
|
21
|
0
|
-35
|
16
|
51
|
14
|
12/13/2015
|
Cowboys
|
@
|
Packers
|
0.459
|
0
|
-1
|
22
|
24
|
0
|
-21
|
7
|
28
|
14
|
12/13/2015
|
Eagles
|
Bills
|
0.423
|
0
|
2
|
23
|
25
|
1
|
3
|
23
|
20
| |
14
|
12/13/2015
|
Falcons
|
@
|
Panthers
|
0.433
|
0
|
-3
|
24
|
24
|
0
|
-38
|
0
|
38
|
14
|
12/13/2015
|
Jaguars
|
Colts
|
0.382
|
0
|
-3
|
21
|
25
|
1
|
35
|
51
|
16
| |
14
|
12/13/2015
|
Jets
|
Titans
|
0.564
|
1
|
-3
|
20
|
21
|
1
|
22
|
30
|
8
| |
14
|
12/13/2015
|
Lions
|
@
|
Rams
|
0.788
|
1
|
14
|
26
|
12
|
0
|
-7
|
14
|
21
|
14
|
12/13/2015
|
Packers
|
Cowboys
|
0.541
|
1
|
1
|
24
|
22
|
1
|
21
|
28
|
7
| |
14
|
12/13/2015
|
Panthers
|
Falcons
|
0.567
|
1
|
3
|
24
|
24
|
1
|
38
|
38
|
0
| |
14
|
12/13/2015
|
Patriots
|
@
|
Texans
|
0.695
|
1
|
3
|
25
|
23
|
1
|
21
|
27
|
6
|
14
|
12/13/2015
|
Raiders
|
@
|
Broncos
|
0.521
|
1
|
-1
|
21
|
22
|
1
|
3
|
15
|
12
|
14
|
12/13/2015
|
Rams
|
Lions
|
0.212
|
0
|
-14
|
12
|
26
|
1
|
7
|
21
|
14
| |
14
|
12/13/2015
|
Ravens
|
Seahawks
|
0.328
|
0
|
-5
|
19
|
23
|
0
|
-29
|
6
|
35
| |
14
|
12/13/2015
|
Redskins
|
@
|
Bears
|
0.301
|
0
|
-3
|
24
|
28
|
1
|
3
|
24
|
21
|
14
|
12/13/2015
|
Saints
|
@
|
Buccaneers
|
0.518
|
1
|
1
|
27
|
26
|
1
|
7
|
24
|
17
|
14
|
12/13/2015
|
Seahawks
|
@
|
Ravens
|
0.672
|
1
|
5
|
23
|
19
|
1
|
29
|
35
|
6
|
14
|
12/13/2015
|
Steelers
|
@
|
Bengals
|
0.404
|
0
|
-2
|
24
|
26
|
1
|
13
|
33
|
20
|
14
|
12/13/2015
|
Texans
|
Patriots
|
0.305
|
0
|
-3
|
23
|
25
|
0
|
-21
|
6
|
27
| |
14
|
12/13/2015
|
Titans
|
@
|
Jets
|
0.436
|
0
|
3
|
21
|
20
|
0
|
-22
|
8
|
30
|
14
|
12/14/2015
|
Dolphins
|
Giants
|
0.608
|
1
|
2
|
26
|
22
|
0
|
-7
|
24
|
31
| |
14
|
12/14/2015
|
Giants
|
@
|
Dolphins
|
0.392
|
0
|
-2
|
22
|
26
|
1
|
7
|
31
|
24
|
No comments:
Post a Comment