2017 NFL Game Predictions: Rest of Season, Playoffs, Best Team

 Welcome!  I intend this to be an ongoing project of predicting NFL game outcomes, point spreads, and final scores.  My hope is that my models will improve over time as they become more sophisticated and use better data.  I will try to regularly publish predictions to keep myself accountable, and will make observations along the way about what is working and what isn't.  See below for latest updates.  Enjoy!

  _________________________________________________________________________________

Welcome!  In years past I have predicted NFL scores and outcomes through the season and into the playoffs and Superbowl.  With work, school, and family life, this was not possible this year.  However, I managed to scrap together a little time to run the code to see what the rest of the season looked like.  With a few minor changes to the code, I was able to successfully run predictions for week 16, 17, the playoffs, Superbowl, and a "best" team prediction.  Given that I may not have time to revisit this, I lay out the predictions for the rest of the season below.

Week 16:

Not having followed much of the NFL this year, I am not sure I can really comment on the below games.  But from what little I know (a quick sanity check using win/loss comparisons), the below outcome predictions seem reasonable, although some of the probabilities strike me as a little strange.

Week	Date	Team	HomeAway	Opponent	ProbabilityWin	PredictedTeamWin
16	12/23/2017	Colts	@	Ravens	0.133	0
16	12/23/2017	Packers		Vikings	0.211	0
16	12/23/2017	Ravens		Colts	0.867	1
16	12/23/2017	Vikings	@	Packers	0.789	1
16	12/24/2017	49ers		Jaguars	0.406	0
16	12/24/2017	Bears		Browns	0.727	1
16	12/24/2017	Bengals		Lions	0.295	0
16	12/24/2017	Bills	@	Patriots	0.371	0
16	12/24/2017	Broncos	@	Redskins	0.257	0
16	12/24/2017	Browns	@	Bears	0.273	0
16	12/24/2017	Buccaneers	@	Panthers	0.075	0
16	12/24/2017	Cardinals		Giants	0.866	1
16	12/24/2017	Chargers	@	Jets	0.774	1
16	12/24/2017	Chiefs		Dolphins	0.628	1
16	12/24/2017	Cowboys		Seahawks	0.399	0
16	12/24/2017	Dolphins	@	Chiefs	0.372	0
16	12/24/2017	Falcons	@	Saints	0.447	0
16	12/24/2017	Giants	@	Cardinals	0.134	0
16	12/24/2017	Jaguars	@	49ers	0.594	1
16	12/24/2017	Jets		Chargers	0.226	0
16	12/24/2017	Lions	@	Bengals	0.705	1
16	12/24/2017	Panthers		Buccaneers	0.925	1
16	12/24/2017	Patriots		Bills	0.629	1
16	12/24/2017	Rams	@	Titans	0.581	1
16	12/24/2017	Redskins		Broncos	0.743	1
16	12/24/2017	Saints		Falcons	0.553	1
16	12/24/2017	Seahawks	@	Cowboys	0.601	1
16	12/24/2017	Titans		Rams	0.419	0
16	12/25/2017	Eagles		Raiders	0.813	1
16	12/25/2017	Raiders	@	Eagles	0.187	0
16	12/25/2017	Steelers	@	Texans	0.948	1
16	12/25/2017	Texans		Steelers	0.052	0

Week 16 Results:

• Estimated: 11.743 (12) correct, so 4 incorrect
• Actual: 14 - 2
• Jaguars lost to the 49ers
• Bengals beat the Lions
• Comments
• Seahawks may actually make it!  I have the Falcon's losing in week 17 and the Seahawks winning.  If both happen, Seahawks make the wildcard spot!
• I have the Titans losing to the Jaguars, so they would go 8-8.  Chargers will beat the Raiders, so they will go 9-7.  Bills will beat the Dolphins so they will go 9-7.  Since the Chargers beat the Bills earlier this season, I'd now pick the Chargers to get into the playoffs over the Bills.
• I'm a little shaken in my confidence in the Jaguars to be the team to win the Superbowl.  The 49ers are looking really good though and are a different team with Jimmy Garoppolo, having won 4 in a row since he became the starter.  So perhaps it is not the end of the world for my audacious predictions.

Week 17:

What about week 17?  Again, hard to comment since I haven't been following closely.  But the below again looks reasonable.

Week	Date	Team	HomeAway	Opponent	ProbabilityWin	PredictedTeamWin
17	12/31/2017	49ers	@	Rams	0.416	0
17	12/31/2017	Bears	@	Vikings	0.233	0
17	12/31/2017	Bengals	@	Ravens	0.375	0
17	12/31/2017	Bills	@	Dolphins	0.652	1
17	12/31/2017	Broncos		Chiefs	0.129	0
17	12/31/2017	Browns	@	Steelers	0.072	0
17	12/31/2017	Buccaneers		Saints	0.089	0
17	12/31/2017	Cardinals	@	Seahawks	0.259	0
17	12/31/2017	Chargers		Raiders	0.817	1
17	12/31/2017	Chiefs	@	Broncos	0.871	1
17	12/31/2017	Colts		Texans	0.473	0
17	12/31/2017	Cowboys	@	Eagles	0.338	0
17	12/31/2017	Dolphins		Bills	0.348	0
17	12/31/2017	Eagles		Cowboys	0.662	1
17	12/31/2017	Falcons		Panthers	0.419	0
17	12/31/2017	Giants		Redskins	0.093	0
17	12/31/2017	Jaguars	@	Titans	0.591	1
17	12/31/2017	Jets	@	Patriots	0.101	0
17	12/31/2017	Lions		Packers	0.737	1
17	12/31/2017	Packers	@	Lions	0.263	0
17	12/31/2017	Panthers	@	Falcons	0.581	1
17	12/31/2017	Patriots		Jets	0.899	1
17	12/31/2017	Raiders	@	Chargers	0.183	0
17	12/31/2017	Rams		49ers	0.584	1
17	12/31/2017	Ravens		Bengals	0.625	1
17	12/31/2017	Redskins	@	Giants	0.907	1
17	12/31/2017	Saints	@	Buccaneers	0.911	1
17	12/31/2017	Seahawks		Cardinals	0.741	1
17	12/31/2017	Steelers		Browns	0.928	1
17	12/31/2017	Texans	@	Colts	0.527	1
17	12/31/2017	Titans		Jaguars	0.409	0
17	12/31/2017	Vikings		Bears	0.767	1

Week 17 Results:

• Estimated:  11.8 (so expected 12 correct, 4 incorrect)
• Actual: 7-9
• Comments
• I really need to look at what happens when a team has secured a playoff spot and has nothing to gain by winning their last game.  It makes sense for a team in this position to not make an effort to win the last game, to rest their players, avoid injuries, and prepare for the playoffs.  Although I haven't watched the replays, I suspect this can help explain why the Eagles, Jaguars, Panthers, Rams, and Saints all lost.  Maybe next year I can adjust my model to account for this factor.
• Falcons won and the Seahawks lost, so the Falcons made the playoffs after all.  The missed field goal at the end of the game is, in a way, a summary of the Seahawks entire season: at times, greatness, but too often, a miss.
• In the AFC, the Ravens are out and the Titans are in the playoffs.  Bills made it in over the Chargers.  Lots of tie breaking rules at play here.  Four teams tied at 9-7: Titans, Ravens, Chargers, and Bills.  See here for how this tie breaking worked out.
• In the NFC, all of the teams made it that I thought would, although the Rams are seeded at 3 and the Saints are seeded at 4.  Panthers, Saints, and Rams tied at 11-5.  The Rams beat the Saints earlier in the season so they go ahead of the Saints, and the Saints beat the Panthers, so they go ahead of the Panthers.
• Playoff impact?
• Titans vs. Chiefs - my model picks the Titans to win and the Chief's to lose.  Titan's will lose to Patriots, so that doesn't change.
• Rams vs. Falcons-Rams beat Falcons and then play Vikings but lose to Vikings, so the Vikings continue.
• Panthers vs. Saints - my model predicts a Panther's victory.  Panthers play the Eagles and lose, so the Eagles go on.
• So while things are initially different, the ultimate outcome is not changed (based on my model from two weeks ago, which should probably be updated).

Playoffs and Superbowl

So what if the above is mostly correct?  Who will get into the playoffs?  My current predictions suggest the following by seed:

• AFC - Patriots (1), Steelers (2),  Jaguars (3), Chiefs (4), Ravens (5), Bills (6)
• Patriots beat the Steelers, so in an expected tie in record, Patriots get the first seed.
• The Chiefs beat the Chargers twice, so in the case of a tie, they get the division. 
• Ravens will likely go 10-6, so would have the top wildcard spot.
• The Bills lost to the Chargers, so if they tie, the Chargers would go in ahead of them.  Neither of them has played the Titans. Bills have a slightly higher probability of winning, so I am giving them the 6th seed, although it could be the Chargers or Titans.
• NFC - Eagles (1), Vikings (2), Saints (3), Rams (4), Panthers (5), Falcons (6)
• Eagles will likely go 14-2 and get the top spot, with Vikings second.
• Saints beat the Panthers twice and I expect will go 12-4 so I gave them the third seed.  I also expect the Panthers to go 12-4, but Saints would win in the tie break.
• Rams should win the West and get the 4 seed.
• Panthers then get the highest wildcard (5), and the Falcons get (6).

Sadly (at least for me), it looks like the Seahawks are out of the playoffs. But clearly, a lot can still change...

 

Team	Remaining Predicted Wins	Remaining Predicted Losses	Previous Wins	Previous Losses	Total Wins	Total Losses	Round Wins	Round Losses	Conference	Division	Rank Conference
Patriots	1.528	0.472	11	3	12.528	3.472	13	3	AFC	East	1
Steelers	1.876	0.124	11	3	12.876	3.124	13	3	AFC	North	2
Jaguars	1.185	0.815	10	4	11.185	4.815	11	5	AFC	South	3
Ravens	1.492	0.508	8	6	9.492	6.508	9	7	AFC	North	4
Chiefs	1.499	0.501	8	6	9.499	6.501	9	7	AFC	West	5
Bills	1.023	0.977	8	6	9.023	6.977	9	7	AFC	East	6
Titans	0.828	1.172	8	6	8.828	7.172	9	7	AFC	South	7
Chargers	1.591	0.409	7	7	8.591	7.409	9	7	AFC	West	8
Dolphins	0.72	1.28	6	8	6.72	9.28	7	9	AFC	East	9
Raiders	0.37	1.63	6	8	6.37	9.63	6	10	AFC	West	10
Bengals	0.67	1.33	5	9	5.67	10.33	6	10	AFC	North	11
Broncos	0.386	1.614	5	9	5.386	10.614	5	11	AFC	West	12
Jets	0.327	1.673	5	9	5.327	10.673	5	11	AFC	East	13
Texans	0.579	1.421	4	10	4.579	11.421	5	11	AFC	South	14
Colts	0.606	1.394	3	11	3.606	12.394	4	12	AFC	South	15
Browns	0.345	1.655	0	14	0.345	15.655	0	16	AFC	North	16
Eagles	1.475	0.525	12	2	13.475	2.525	13	3	NFC	East	1
Vikings	1.556	0.444	11	3	12.556	3.444	13	3	NFC	North	2
Saints	1.464	0.536	10	4	11.464	4.536	11	5	NFC	South	3
Panthers	1.506	0.494	10	4	11.506	4.494	12	4	NFC	South	4
Rams	1.165	0.835	10	4	11.165	4.835	11	5	NFC	West	5
Falcons	0.866	1.134	9	5	9.866	6.134	10	6	NFC	South	6
Seahawks	1.342	0.658	8	6	9.342	6.658	9	7	NFC	West	7
Lions	1.442	0.558	8	6	9.442	6.558	9	7	NFC	North	8
Cowboys	0.737	1.263	8	6	8.737	7.263	9	7	NFC	East	9
Packers	0.474	1.526	7	7	7.474	8.526	7	9	NFC	North	10
Redskins	1.65	0.35	6	8	7.65	8.35	8	8	NFC	East	11
Cardinals	1.125	0.875	6	8	7.125	8.875	7	9	NFC	West	12
Bears	0.96	1.04	4	10	4.96	11.04	5	11	NFC	North	13
49ers	0.822	1.178	4	10	4.822	11.178	5	11	NFC	West	14
Buccaneers	0.164	1.836	4	10	4.164	11.836	4	12	NFC	South	15
Giants	0.227	1.773	2	12	2.227	13.773	2	14	NFC	East	16

Suppose that the above playoff spot predictions are true.  Then what happens?  Using head to head predictions based on the season so far:

Wildcard:

Jaguars (3) vs. Bills (6) -> Jaguars (3) (0.569)

Chiefs (4) vs. Ravens (5) -> Ravens (5) (0.529)

Saints (3) vs. Falcons (6) -> Saints (3) (0.55)
Rams (4) vs. Panthers (5) -> Rams (4) (0.501)

Division:

Patriots (1) vs. Ravens (5) -> Patriots (1) (0.71)
Steelers (2) vs. Jaguars (3) -> Jaguars (3) (0.523)

Eagles (1) vs. Rams (4) -> Eagles (1) (0.501)
Vikings (2) vs. Saints (3) -> Vikings (2) (0.514)

Conference:
Patriots (1) vs. Jaguars (3) -> Jaguars (3) (0.542)
Eagles (1) vs. Vikings (2) -> Vikings (2) (0.501)

Superbowl:

Vikings (2) vs. Jaguars (3) -> Jaguars (3) (0.509)

Shocking!  The Jaguars will beat the Steelers, then the Patriots, before beating the Vikings in the Superbowl!  I should note that I originally had the Eagles going all the way and winning the Superbowl, but after factoring in injuries into my model, this was enough to push the Eagles out.  It's too bad Carson Wentz is injured, otherwise I would have the Eagles going all the way.

Best Team of 2017

If each team played every other team, who would come out on top?  If measured by total probability of wins, the Patriots, Vikings, and Steelers take 1, 2, and 3.  The Jaguars come in 4th.  The Eagles are down in 9th (before injuries, they were at 3).  The Giants, Texans, and Buccaneers, take 32,31, and 30.  If measured by total games won, then the Jaguars, Vikings, and Eagles take 1, 2, and 3, while the Giants, Colts, and Buccaneers take 32,31, and 30. The Patriots drop to number 7.

What does this mean?  In short, it means that the Patriots have the highest total probability of winning each game when they are predicted to win.  However, they are predicted to win fewer times than other teams.  As such, they are less consistent, prone to have a high probability of winning one week but then losing the next week by a narrow probability.  Meanwhile, for example, the Vikings are the only team to appear in both top three lists, meaning that they are expected to win with a fairly high probability when they are predicted to win AND predicted to win every game.  However, their total probabilities are lower than the Patriots, meaning that they are predicted to have closer victories than the Patriots, but more victories in total.

I should note that before factoring in injuries, the Eagles were my top choice.  But after injuries and the loss of their QB, their ranking has significantly fallen.

The Vikings appear in both lists at 2.  Jaguars are at 4 and 1.  Patriots are at 1 and 9.  Eagles are at 9 and 3.  Steelers are at 3 and 6.  Any of these teams has a claim at being considered the best team of the season.  Originally, I would have said the Eagles, as I had predicted them to win the Superbowl and they appeared in both top 3 lists.  After factoring in injuries, total probability of winning, and total games predicted to win, I am inclined to pick the Jaguars with the Vikings barely behind.  If the Jaguars can beat the Patriots, Steelers, and then beat the Vikings in the Superbowl as I have predicted, they will without a doubt be the best team of the year.  Thus, I conclude that the Jaguars are (or will be) the best team of the year.

Total Probability:

  

Rank	Team	Expected Value
1	Patriots	23.376
2	Vikings	22.234
3	Steelers	22.124
4	Jaguars	21.967
5	Panthers	21.914
6	Rams	21.623
7	Saints	21.236
8	Falcons	21.046
9	Eagles	20.896
10	Chargers	20.286
11	Lions	19.844
12	Seahawks	19.799
13	Bills	19.246
14	Titans	19.121
15	49ers	18.623
16	Ravens	18.326
17	Cowboys	17.831
18	Chiefs	17.524
19	Dolphins	15.069
20	Bengals	15.046
21	Packers	14.251
22	Redskins	14.011
23	Cardinals	13.375
24	Bears	13.13
25	Raiders	11.364
26	Jets	10.437
27	Broncos	8.201
28	Browns	6.772
29	Colts	6.725
30	Buccaneers	6.511
31	Texans	6.315
32	Giants	3.777

Total Wins:

Rank	Team	Total Wins
1	Jaguars	32
2	Vikings	31
3	Eagles	30
4	Rams	29
5	Panthers	28
6	Steelers	27
7	Patriots	26
8	Saints	25
9	Falcons	24
10	Chargers	23
11	Lions	22
12	Seahawks	21
13	Bills	20
14	Titans	19
15	49ers	18
16	Ravens	17
17	Cowboys	16
18	Chiefs	15
19	Dolphins	14
20	Bengals	13
21	Packers	12
22	Redskins	11
23	Cardinals	10
24	Bears	9
25	Raiders	8
26	Jets	7
27	Broncos	6
28	Texans	5
29	Browns	4
30	Buccaneers	3
31	Colts	2
32	Giants	1

Averages:

Team	AverageRank	AverageWins
Jaguars	2.5	26.9835
Vikings	2	26.617
Eagles	6	25.448
Rams	5	25.3115
Panthers	5	24.957
Patriots	4	24.688
Steelers	4.5	24.562
Saints	7.5	23.118
Falcons	8.5	22.523
Chargers	10	21.643
Lions	11	20.922
Seahawks	12	20.3995
Bills	13	19.623
Titans	14	19.0605
49ers	15	18.3115
Ravens	16	17.663
Cowboys	17	16.9155
Chiefs	18	16.262
Dolphins	19	14.5345
Bengals	20	14.023
Packers	21	13.1255
Redskins	22	12.5055
Cardinals	23	11.6875
Bears	24	11.065
Raiders	25	9.682
Jets	26	8.7185
Broncos	27	7.1005
Texans	29.5	5.6575
Browns	28.5	5.386
Buccaneers	30	4.7555
Colts	30	4.3625
Giants	32	2.3885

Conclusion

So there you have it.  If you haven't been paying attention all season, now having read this post, you can speak somewhat accurately about which teams are good, bad, and likely to make or do well in the playoffs.  Just make sure no one asks you why...

Good luck!

Data Science and Philosophy of Science: Theory and Reality

Introduction

A major branch of philosophy called metaphysics may be characterized as the study of "being" or the nature of reality.  A sub-branch of metaphysics called ontology focuses on what exists and categories of existence.  The nature of causation is also considered within the realm of metaphysics.  Related to (and perhaps a further sub-branch of) these areas of philosophy is the philosophy of science, which studies the disciplines of science to understand their processes, assumptions, and commitments from a philosophical standpoint.

How might metaphysics and its related sub-branches/topics apply to the field of data science?  While there are several possibilities, I will focus on a debate in philosophy of science regarding the ontological status of scientific theories and the objects that are part of these theories, before applying that debate to data science.

Why is Science Successful? Realism vs. Instrumentalism vs. Constructive Empiricism

Why is science successful?  What needs explaining is the relationship between our scientific theories and objective reality.  For example, when we say that the Earth revolves around the sun, what does that mean?  Does that mean that we are claiming that the sun, the Earth, and the various gravitational forces actually exist in a mind independent way?  Or do we just mean that these concepts or ways of thinking are useful, practical, and help us achieve the ends we have, but that we shouldn't place any weight on the existence of such objects? 

This is the debate between scientific realism and instrumentalism (and constructive empiricism).  While the answer seems easy when discussing objects that we can verify with our senses, the answer is much more difficult when it comes to so called unobservable entities.  Does dark matter or anti-matter exist?  How do we know?  What about the theory of quantum mechanics?  Is a true depiction of physical reality, or is it just a useful way of thinking that makes the math come out properly?

Let's explore what each view is and what arguments for and against it there are.  For you hard-core philosophy of scientists, please forgive me if I am not exactly correct in my discussion below :)

Realism

Scientific realism is the view that "science aims to give us, in its theories, a literally true story of what the world is like; and acceptance of a scientific theory involves the belief that it is true" (Bas van Fraassen in The Scientific Image (1980)).  The intuition behind scientific realism is that we have made significant progress using scientific theories, and as science progresses, the theories we have are more reliable and lead to more reliable predictions.  The best way to account for this is that our scientific theories are true.  Hence, the objects of our theories we ought to consider to be real. 

So even if we have to postulate unobservable entities, we should believe these to be real and existing, even if we cannot see them, because what we can see and observe implies their existence.  Hence, unobservable entities are "discovered" in a sense, and not simply created.  The unobservable entities are considered to be mind independent (and theory independent).  Granted, science is fallible and theories are approximate, but we should think about scientific theories as being approximately true.

Instrumentalism

But what about the objects of scientific theories that are no longer accepted?  What about ether or ectoplasm?  These are things that were postulated to scientifically explain phenomena, and they may have worked well, but they have been discovered to not exist, and more reliable theories with different postulated objects have replaced them.  How can we be certain that unobservable entities in our current scientific theories will not suffer the same fate?  What about various geocentric models for the solar system that, although considered incorrect now, were still highly accurate in predicting phenomena?  How can we be certain that our current theories will survive the test of time?

This is the intuition behind instrumentalism.  Scientific theories "are just conceptual tools for classifying, systematizing, and predicting observational statements" (Duhem in The Aim and Structure of Physical Theory (1954)). This is pragmatism as applied to scientific theory.  Our scientific theories help us be more reliably successful in prediction and empirical observation, and they certainly can explain things, but we shouldn't take the things they talk about to be real or true, and we can't apply these theories to new issues.  In fact, the scientific statements we make cannot be considered to be true or false: they are just useful, and some are more useful than others.

Constructive Empiricism

Constructive empiricism may be thought of as a middle ground between realism and instrumentalism.  The constructive empiricist says that "science aims to give us theories which are empirically adequate; and acceptance of a theory involves as belief only that it is empirically adequate"  (Bas van Fraassen in The Scientific Image (1980)).  What does this mean?  In short, in contrast to instrumentalism, we can say that scientific statements can be true or false.  However, this does not commit us to believing in the unobservable entities.  In fact, one's acceptance of a theory is more of a commitment to continue to do research within that framework and to use the framework for new issues.

Arguments For and Against

All three views agree that a good theory must be predictive, informative, simple, and explanatory to be good (I'll discuss why in a follow up post).  However, they disagree on what this implies ontologically speaking.  So why believe one view over another?

In support of realism, we might regard the success of science as being miraculous if the objects in our theories are not real, that is, there is no relationship to or correspondence with reality.  It seems that the terms in our theories must refer to something real, although perhaps our characterization of those things is only approximately correct.  Furthermore, what is observable vs. unobservable is always changing, and what was once unobservable has become observable (e.g., bacteria, protons).  So even if something is unobservable now, it doesn't mean that is in principle unobservable or will never become observable.

But the anti-realist will respond that two theories can be equivalent empirically and yet postulate different unobservables, so they can't both be correct.  Both are empirically adequate.  Perhaps we should just say that neither is "true" though both are useful.

While I obviously won't resolve this debate, here are some quick thoughts.  It seems we should take some unobservables to be real, because past unobservables have become observable or have been disproven to exist.  Second, it seems scientific theories must be capable of being true or false to avoid pseudoscientific claims that cannot be disproven (I'll table this for a follow up post), and that when we speak about scientific theories, we are making claims about how the world really is.  Third, there may be unobservable entities that nevertheless exist.  Just because we cannot detect something does not mean it does not exist (this is to fall back into logical positivism).  This may make a theory that contains these difficult or perhaps physically impossible to falsify, but so long as it could in principle be falsified, perhaps this is ok.  So call me a cautious realist.

So what does this have to do with data science?

Why is Data Science Successful?

"All models are wrong, but some are useful."  So says George Box, a famous statistician, and this is something always quoted in any introduction to data science.  Which view does this match up with?  This seems to be an anti-realist position.  Is he correct?

Let's compare scientific models to the typical data science model from the anti-realist perspective.  First, the objects typically in scientific models are physical things (objects, forces), whereas the variables in a typical data science model are mostly conceptual.  We try to predict things like prices or movie ratings using variables like age or genre.  These are not objects in the world.  Instead, they are usually ways of expressing a value that people have. 

Second, scientific models always assume a causal relationship of some kind, whereas typical data science models don't postulate any sort of causal relationship, only correlation (and we all know, correlation does not equal causation).  How would the age of a person cause a certain movie rating, anyway?

Third, the relationship between the objects described in a scientific model is supposed to be unchanging, a law, whereas the relationships in typical data science models always seems to be changing and also depends on which variables are being considered.  For example, the coefficient for a certain variable may completely change depending on which other variables are being included in a model.

I could go on, but how might the realist respond to these objections?  First, one might say that our variables are short hand or abbreviations for physical things.  For example, a movie rating could be a proxy for something physical, like the amount of dopamine triggered in a person's brain when viewing that movie, and the genre composed of the various kinds of images and sounds depicted in the movie.  These are all physical things, but it is more useful and practical to speak about them in higher-level summary non-physical terms. 

Second, leaving aside debates about what causation actually is, using the prior response, we could tell a causal story between these variables.  When we say that the "increased square footage of the house caused its price to go up", we are actually saying something like "the amount of dopamine triggered in the brain by the recognition of increased square footage of the house triggered another neuron to fire that led to the individual valuing the house more, and being willing to pay more money for it."  This isn't exactly perfect, but you get the point: we can translate the non-physical entities and forces into physical entities and forces that underlie, compose, and give rise to them. 

Third, because these abbreviations and summaries are imprecise and don't account for every physical and causal entity, it shouldn't surprise us that our data science models are not perfect, or that the introduction of new variables (i.e., physical entities and forces) should change our understanding of the relationships.  However, that does not imply that the relationships are not legitimate or approximately true.  In fact, we have this intuition that if we could account for every physical entity and force in our data science model, then it would in fact be 100% accurate.

Conclusion

So what do I think?  I tend to have realist tendencies, so I am inclined to favor a realist interpretation of data science models.  I believe that our models must be attaching on to something real if they are in fact being useful.  And our concepts do attach to real entities and forces in the world, although they are summaries of very complex relationships among things and perhaps are not real in themselves (that is another debate).  So I am not surprised that our models are only approximately true (i.e., close in predicting numeric values, more often correct than not in categorizing). 

And while two models may both be equal in empirical accuracy, experience suggests that either only a few common variables are doing the main predictive work (and the additional variables are mostly noise) or the differing variables are both correlated with a third variable that really explains the correlation.  With additional work and time, the two supposedly differing models should converge and begin to look more and more similar as predictive accuracy increases.  Again, this suggests that there is something real that the models are latching on to, although it may be disguised and hard to uncover in a precise manner.

So I would respond that, strictly speaking, Box is correct: if a model is either all true or all false, then all models are in fact false.  But if models can be approximately true, then some models are approximately truer than others, and in this sense, are modeling reality and the relationships among entities in a truer and more real (and perhaps causal) way.

So how can we make sure that the data science model we have is approximating reality?  We'll explore this is a follow up post.

Predicting House Prices

Introduction

We all know that when it comes to housing prices, the most important thing is location, location, location.  But what else matters?  A Kaggle competition asks that very question (see here for details).  Using data from home sales in Ames, Iowa, the competition asks us to predict the final sales price.  Variables to use in that prediction include features such as year built, square footage, number of bathrooms, number of bedrooms, central heating, and other potentially important factors.

Analysis

So what does matter?  I entered the competition as part of a final project for a class in my data science Master's program.  My full process and results as part of that class are located here for you take a look at:

• All files: https://github.com/anrcarson/CUNY-MSDA/tree/master/DATA605
• RMarkdown: http://rpubs.com/anrcarson/339773
• Youtube video presentation: https://youtu.be/BXLST5Ls5Os

But here is quick summary if you want the short version of the results:

• My best model had an R^2 value of 0.935.  While this sounds great, it only got me barely into the top 50% in rank.  Kaggle competitions are hard.
• The most important variables (according to my models) in predicting sales price are:
• how much square footage does the house have? More is better.
• what kind of roof does it have? Certain kinds (mebrane, metal) are better than others (Composite Shingle, Woodshake).
• what is the overall condition of the house? A higher ranking is better.
• zoning (commercial zoning lowers the price)
• overall quality (higher is better)
• basement square footage (more is better)
• lot area (more is better)

If I had more time to complete my analysis, what else would I do?  At the very least, I would try to do more feature engineering and would break the given training data into training and testing sets so that I could test my model generalization and correct overfitting before submitting my predictions to Kaggle.  I would also normalize the variables so that I could better understand the impact of each variable, comparing coefficients to see which variables contribute the most to the overall price, and not just which ones correlate the most.

But with limited time and an open ended problem to "solve", one must draw a line somewhere and say "good enough" to avoid increasingly diminishing returns on time and effort.  And since my solution was "good enough" to help me really understand the nature of the problem and to complete my final project, I left my model as is after breaking the 50% mark.

Predicting My House

Most of you will know that some websites already provide an estimate of how much your house is worth based on factors such as those considered in the Kaggle competition (e.g., Zillow's Z-estimate).  In August 2010, my house was listed for $300K by the previous owner.  As I live in the Seattle, WA housing market, prices are definitely NOT typical compared to other places in the US (and have only gotten worse since 2010, or better depending on your point of view).

What would my model predict the SalePrice of my house to be?  My house is NOT in Ames, Iowa, so the most important factor in determining house price (location, location, location) is different and is not being accounted for.  Still, I was curious, so I filled in all of the variables for my house (e.g., square footage) to see what my model predicted my house would have been sold for if it had been in Ames, Iowa in 2010.

After filling in the data for my house, my model predicted a value of $168,455.  When I manually looked at the training data for similar houses to mine, the average price was comparable ($160K-$180K).  So my model seems to be predicting close to what I would expect for my house IF it were in Ames.  I take the $130K difference therefore to be the price due to a difference in location, location, location at that time.  I'm sure that this difference is now much greater.

Conclusion

None of what my analysis revealed is really surprising.  The factors that are important are things I would consider important when looking at a house.  Perhaps the Kaggle competition is best understood as being less about understanding what is important and more about understanding and quantifying how important these things are.  That is, which factors contribute the most value to the price of a house?

While I would need to do more work to more accurately assess the magnitude of impact that each variable has in determining the house price, a quick multiplication of the coefficient for a variable in the model by the average value of the variable can tell us roughly what the average impact of each variable is in contributing towards the final sales price.

The SalePrice for a house using average values and the coefficients in the model is about $160K.  Here are some of the most important variables  that make up the following percentage contributions towards that price:

• Intercept: 59% (starting value for a house)
• Roof Material - Composite Shingle: 23%
• LotArea: 7.3%
• 1st floor square footage: 1.9%
• YearsOld: -4.0%

Composite shingle is the most common roof material, so this isn't really that interesting (one can think of this as generally adding to the intercept except in a few cases).  Unsurprisingly, lot area and square footage are important contributions towards adding value to the sale price, while years old removes value from the sale price. 

Again, not very surprising, although the fact that lot area appears to matter more than square footage in determining price is perhaps a little surprising.  But if one remembers that location, location, location is what matters most, then it makes sense that how much you own of that location, location, location may be more important than what is on that location, location, location.

To conclude, I hope that the above summary gives you some insight into what makes a house valuable and why that might be the case.  I believe my analysis matches our intuitions about what determines a sales price for a house, so if nothing else, it confirms what we already intuitively know, and gives us a way to quantify those intuitions.

Thanks!