Classifying NFL Team Logos with Tensorflow Keras

Introduction

In fulfilling the final project requirements for a class on "Big Data and Machine Learning", which is part of my continued MS Data Science studies, the below report was created.  Some modifications have been made, but the substance remains largely the same.

The original report is here: https://github.com/anrcarson/CUNY-MSDA/blob/master/DATA622/finalproject-anrcarson/Report.md

The Task

Build an image recognition model and summarize your results. You may choose any data set that you are interested in exploring.

The Solution

Overview

For my project I downloaded images of NFL logos for the four following teams that make up the NFC West: Cardinals, Rams, Seahawks, and 49ers.  My task was then to build an image classifier that correctly classified each logo image to the corresponding team.  I followed TensorFlow Keras tutorials (relying most heavily on the link here: https://www.tensorflow.org/tutorials/images/classification), as well as code snippets from around the internet, to build an end-to-end solution.  The final code (imageClassification.ipynb) ran on my personal machine.

Here are the files/folders contained in my final GitHub repository (link is here: https://github.com/anrcarson/CUNY-MSDA/tree/master/DATA622/finalproject-anrcarson):

• ModelImages: the results of downloadpics.py (the code used to search for and download images).  Sorted into train/test sets.
• Screenshots: screenshots of my progress along the way.  Images referenced below.
• README.md: initial instructions as well as additional instructions for the project.
• Report.md: this file.  The final report.
• downloadpics.py: the code used to leverage the Bing Image Search API to find and download images of NFL team logos for use in building the model.
• imageClassification.ipynb: the image processing and model building, training, and testing code.  

Image Search

I used the Bing Image Search API, available via Azure Cognitive Services, to search for images relating to NFL team logos.  After signing up for Azure and setting up the API, I used the key provided along with sample code to do my automated searching. For example, one search was for "seahawks logo", just as you would do an image search in Bing using a browser.  The results were then paged through, while each main image url was used for downloading the image locally for later use in training the model.

Not all images were useful or accurate for my purposes.  I manually combed through the roughly 800-900 images returned for each of the four teams to only retain those that I judged to accurately match the team in question.  I also needed to remove animated GIFs that created errors in code processing to keep only JPEGs.  This left about 600-700 images per team for a total of 2653 images.

Split into Train and Validation

Having gotten the images, I needed to split them into a training set and validation set for use in training the model.  I created some code to do this automatically using a 67% training 33% validation split.  The code pulled evenly from each team class.  This gave me 1765 training images and 888 validation images.  In exploring documentation for something else I learned that I could have used the "validation_split" in the image_generator to do this for me, but as this was no longer needed, I did not redo this (although I will redo this in the future if I decide to extend this project).

Training

I did several rounds of training using different methods for preventing overfitting, adding additional layers in the model, and using different kinds of image augmentation.  In particular, I experimented with the following:

• With and without data augmentation: image rotation, width shift, height shift, horizontal flip, and zoom. (see "trainingImages" and "trainingImages_withDataAug" screenshots for difference below).
• Without augmentation:
• 
•  With augmentation:
• 
• With and without dropout layers
• With and without learning rate schedules
• With and without early stopping
• With and without L2 regularization

The basic model was a convolutional neural network (CNN).  Using the example of the tensorflow tutorial, I used a sequential model with three Conv2D layer / MaxPooling2D layer combinations (using ReLu activation) before a flatten/dense layer, with a final dense layer of 4 softmax outputs.

Results

A brief history of the results is as follows:

• The initial model with 15 epochs had a training accuracy of 0.99 with a validation accuracy of 0.89.  Not bad, but quite a gap between training and validation.(results_01_originalCNN)

• Adding dropouts lowered the training accuracy to 0.92 and increased the validation accuracy to 0.85.  While decreasing the accuracy this did bring the training closer to the testing accuracy, avoiding overfitting to some degree. (results_02_addDropouts)

• Increasing the epochs to 25 increased the training accuracy back up to 0.99 and the validation accuracy up to 0.89.  No real improvement except that having more epochs gave the model more time to learn. (Note: I think this run may have started on an already trained model, hence the immediate high level of accuracy.) (results_03_25Epochs)

• Adding all of the above data augmentation with 15 epochs lowered the training accuracy to 0.69 and the validation accuracy to 0.69.  This brought training and validation accuracy into alignment. (results_04_15Epochs_DataAug)

• I removed the horizontal flip as I didn't think this was fair to the logos (is the 49ers logo correct if it is backwards?) and increased to 25 epochs while implementing early stopping.  Training accuracy improved to 0.74 and validation accuracy to 0.81 (Note: I did not have data augmentation on the validation data, so these were easier to predict).  So a longer training time was important to improving the accuracy. (results_04_25Epochs_RemoveFlip)

• Adding a learning schedule and L2 regularization in one place increased the accuracy to 0.80 and the validation accuracy to 0.84.  These adjustments appeared to help the model learn better.  However, the model did not have to use early stopping, meaning that even 25 epochs was not long enough for training. (results_05_25Epochs_L2_LearningSch)

• I increased to 100 epochs.  The training accuracy went to 0.87 and the validation accuracy went to 0.88.  The model early stopped at 57 epochs.  I saved this model as the "data augmented" version. (Note: I think this run may have also started on an already trained model). (results_06_100Epochs)

• I removed the data augmentation and reran the training model with the other additions in place.  Stopping at 52 epochs, the model had an accuracy of 0.98 and a validation accuracy of 0.90.  I saved this as the "no data augmentation" version of the model. (results_07_100Epochs_NoDataAug)

Prediction and Misclassified Images

To view the misclassified images, it was necessary (in so far as I could find based on documentation and online discussion) to predict the training images with the shuffle flag set to False.  I then converted the probabilities output by the model into the appropriate class value with the highest probability.  Then I matched these with the correct labels. The model with data augmentation using this method predicted 88.7% correctly on the training data (accuracy_modelWithDataAug).

The accuracy when using the "no data augmentation" version of the model and this method was 99.6% on the training images (accuracy_modelWithNoDataAug), meaning that 7 were misclassified.  I viewed these individually.

Six of the seven images (see misclassifiedImages below) were actually Rams images, and one was a 49ers image.

Here is my best guess as to what happened.

• 49ers shirt classified as "Cardinals"/ Rams shirt classified as "Cardinals": if I look at the training data, the Cardinals have about 20 images of shirts with the Cardinal's logo on it.   The 49ers had fewer shirts and these didn't look as similar to the misclassified image (viewed straight on and flat image).  The Rams had only about 5 shirts.  Hence, the model learned that if it is a certain kind of shirt, it is "Cardinals".
• The exception to the above was a Rams shirt that was labelled as "FortyNiners".  This shirt is 3D looking and viewed from an angle.  The 49ers have a lot of these similar images whereas the Rams have very few.
• The Rams skull is labeled as "FortyNiners": The 49ers have three similar skulls (and another less similar one) in the training data while the Rams have one.  So "if it's a skull, its 'FortyNiners'" was the learned rule.
• Rams carpet circle labeled as Seahawks: Not really sure why.  There were other Rams carpet circles that correctly classified.  The Seahawks logo does have similar colors and geometry, and there were lots of round Seahawks logo images in the training set, so perhaps this is why.
• Rams lettering labeled as "FortyNiners":  the colors here are gray and white.  The 49ers do have some images that are similar, but perhaps it is the fact that the 49ers logo is usually an "SF" or says "49ers".  Meanwhile, the Rams logo usually is just the Ram, the horns, or the the Ram along with the spelled out "Rams".  But not really clear why this was misclassified.
• Rams player labeled as "FortyNiners": The 49ers have a lot more images of players than the Rams do.  I think absent a clear logo or symbol (e.g., side view of the helmet), the learned rule was "if a player, it is FortyNiners".

Future Enhancements

There are lots of variables I could still tweak in the model.  I largely left the initial variables in place that the tutorials used because they seemed to be working well in my testing, they presumably worked well for the tutorials, and they seemed to use best practices from the literature I have read for this sort of problem.  In the future I would experiment with different cost or activation functions, adding additional layers, changing the number of nodes within layers, and changing image sizes or batch sizes.  The changes and combinations I could make to the model are virtually infinite, and while I would expect some improvement, I don't know if much would change in terms of accuracy and generalizability with these model-specific tweaks.

However, I think the biggest improvements would come from better data.  I would try to get more data for each of the teams, and I would try to make the images more balanced across teams.  For example, I could try to balance out the number of images of players per team.  I could remove the "skull" images from the data unless there were similar images available for all of the teams.  That is, I would try to make it so that the only real difference among the images considered as a set for each team is the identifiable logo, symbol, etc. for that team.  I would do more experimentation with the data augmentation controls as well in order to make the model more generalizable to new images.  And I would experiment with the automatic splitting of the data into training and validation sets in hopes that the model would become more generalizable (and not just learn what was in a static training set).

Another future enhancement would be to expand the model to include other teams, and eventually all 32 NFL teams.  I would want to get the model performing very solidly on the current scope of teams and would definitely experiment with the tweaks and data augmentation mentioned above before.  But it would be nice to have a model that covers all teams.  This would be a much more extensive data collection effort, but could be a fun continuation of this project in the future.

Conclusion

My goal in this project (apart from satisfying class requirements) was to find a dataset that I took to be interesting and to build an image classification model on it using the latest deep learning packages and techniques.  By gathering data that was "out in the wild" and not prepackaged, gathered, or preprocessed, I am sure that I could use this approach for other image sets that are also not nicely packaged for image classification (e.g., work related images, personal images).   By leveraging TensorFlow Keras, I have utilized one of (if not the) most popular deep learning code package in current use.  And by following TensorFlow tutorials with not an insignificant number of enhancements and changes, I have utilized the most current methods for creating and fitting an image classification model.  In short, I have built a relatively simple but modern end-to-end image classification deep learning model that can be used as a template or extended for additional projects.

Data Science and Philosophy of Science: What Makes a Model Good?

Introduction

In a previous article, I discussed philosophical views on the nature of scientific theories, and applied these discussions to data science models.  I concluded that data science models, the terms they invoke and the relationships they postulate, ought to be considered to correspond to reality in some way.  That is, a model's terms do in fact represent something real in the world (although this may be an abbreviation, summary, or approximation of potentially many real entities).  Similarly, a model's prescribed relationship does represent something real in the world (e.g., a causal relationship amongst the terms in the model, or amongst hidden terms that make up the terms in the model, or....).  While such correspondence may only be approximate and fall far short of 100% perfection and predictive accuracy, nevertheless, it is not merely useful.  It does approximate the truth, or attach to reality, in some albeit imperfect way.

Whether or not you agree, let's move on to another question in the philosophy of science that does not necessarily depend on how you answered the realist/anti-realist debate: what makes a good scientific model?  How does this apply to data science models?  Let's explore some ideas and then summarize at the end.

The Problem of Induction

Induction is the formation of generalizations or laws on the basis of past experience.  We believe that future occurrences will behave like past occurrences, and so on the basis of past occurrences, we can predict future occurrences.  For example, based on past experiences, we believe that we know (and have mathematically formulated a law) such that when billiard ball A hits billiard ball B in a certain way with a certain force in conditions X, Y, Z, etc., then ball A will go in this direction at this speed and ball B will go in that direction at that speed.

However, we have no guarantee that the past will be like the future in most cases, as there are not typically necessary relationships between the objects we are interested in.  It is conceivable, because it is not a matter of logical necessity, that ball A will spontaneously combust, or turn into a carrot, when it hits ball B.  Such a thing has never occurred before, but that doesn't mean it cannot happen.  Such thoughts have caused some people (most famously David Hume) to be skeptical about our ability to acquire knowledge through induction.

And yet, this is precisely what we do in the sciences.  Even in the absence of logical necessity, we believe that we know what will happen to ball A and ball B in these circumstances, and we can reliably predict what does in fact happen with a very small margin of error.  We even go so far as to form a law, a matter of physical necessity, to explain this relationship.

But what do we do in the face of competing "laws" that both explain the data we have?  Which theory do we go with and use for future research and development of theory?  This is the problem of induction.  How can we justify inductive inferences?  That is, how can we make universal or natural law claims based on experience, when so many alternative claims could be postulated?

Falsifiable

Enter Karl Popper.  His goal is to answer the problem of induction and to distinguish true scientific theories from pseudo-scientific theories.  He observes that it is really easy to formulate a theory that explains the known data, since it is done so using that data (hindsight is 20-20).  While this theory may be correct, one can think of many alternative theories that also explain the data.  How can one tell which theory to accept?

Popper answers that each theory must make so called risky predictions, that is predictions which one should expect to be false unless the theory is right.  A theory that is not refutable is merely pseudo-scientific.  Once we have excluded the pseudo-scientific theories and we have competing scientific theories, we can test them on the basis of what each predicts, focusing in particular on where they would disagree in a prediction.  That is, each theory must propose hypotheses that are then empirically tested after the theory has been formulated.

Conclusions are deduced from the theory, and these are then compared against each other to make sure that the theory is internally consistent, externally consistent with other unfalsified theories, and that when it makes a prediction, that prediction is correct.  When a theory fails to predict accurately or is discovered to be inconsistent, it is falsified.  If it is not inconsistent and does accurately predict, it is acceptable for use (although it may be falsified in the future).  In this, Popper proposes a deductive style method of testing.  We deduce in a manner similar to this: if theory A is true, then X must occur.  X did not occur.  Therefore, A is falsified.

In short, "Science in his view is a deductive process in which scientists formulate hypotheses and theories that they test by deriving particular observable consequences. Theories are not confirmed or verified. They may be falsified and rejected or tentatively accepted if corroborated in the absence of falsification by the proper kinds of tests" (Stanford Encyclopedia).   Theories are true so far if they are successful in making predictions and surviving falsification. Theories are judged by the deductive consequences of the hypotheses they make.

So a virtue of a theory is its ability to be falsified.  Theories that make stronger claims are more falsifiable because the predictions they make are bolder, and typically, more informative.

While this is all well and good, we still have a problem: we can have two theories that are both unfalsified and that make different predictions.  Which should we use until those predictions can be tested?  To answer, let's look at some other virtues that make a model good.

 

Elegance and Parsimony

A theory that is more simple is to be preferred over a more complex theory, all else being equal.  Simplicity can refer to both syntactic simplicity (the number of complexity of hypotheses in a theory; it is elegant) and to ontological simplicity (the number and kinds of entities postulated by the theory; it is parsimonious) (Stanford Encyclopedia).  Most well known, Occam's razor asserts that “entities must not be multiplied beyond necessity."

So why should we prefer more elegant and parsimonious theories?  That is, when faced with a choice between two theories that both explain the data equally well, why choose one over the other on the grounds that one is more simple?  To answer, let us consider the field of epistemology, that is, the study of knowledge.  Knowledge is said to consist in having a justified and true belief.  When faced with competing theories, we are asking ourselves which theory we ought to believe to be true, so our focus is on the justification for each theory.  Now we have already said that each theory is consistent with the data, so what other grounds do we have for believing one theory to more likely be true than another?  Which is more justified?

The simpler theory is more likely to be true because of probability.  Each entity in a theory has a probability of existing or having a certain relationship with the other entities.  So the more we multiply the entities and relationships, the more we multiply probabilities, which always being less than 1, lowers the overall probability.  For example, suppose you have a theory with 2 entities postulated versus 3 entities.  If each entity has a probability of existing/having a certain relationship of 0.75, then the former theory has a probability of  (0.75)^2 =  0.56 versus the latter theory of (0.75)^3 = 0.42 of being true.  Probabilistically speaking, you ought to prefer the former theory because it is more likely to be true, and since the theories are otherwise indistinguishable, you have no other reason to prefer the latter theory.

Or returning to epistemology and the notion of justification, you have no reason for choosing a more complex theory over a more simple theory when both are equally explanatory of the data.  Suppose for example that you return home and find that your house has been robbed.  What would you conclude?  You know that at least one person must have robbed your house.  But are you justified in believing that two people robbed your house?  What about an alien from outer space that came and robbed your house?  If you have no reason to believe that more than one person robbed your house (or that an alien robbed your house), then it seems you are not justified in believing so.  Instead, you must hold the theory that only a single robber broke into your house.  This in spite of the fact that two robbers did really break into your house (unknown to you).  That is, you must hold the most simple theory that explains the data to be true in order for that belief to be justified.

Granted, judgements about which theories are more simple, elegant, and parsimonious can be subjective to a degree.  We may have disagreements in certain cases.  However, we all intuitively have some understanding about what we are talking about and can agree on many cases that one theory is simpler than another.

Predictive and "Accurate"

These last three virtues are mentioned in the discussion on falsifiability, but deserve more attention in their own right.  The first is that a model must be predictive.  This is related to being falsifiable, in that a falsifiable theory makes predictions that can be proven to be false.   But we are interested in theories that not only make predictions, but that make accurate predictions.  In Popper's terms, we want theories that are strongly falsifiable and have failed to be falsified.  These are our best theories and we have made lots of relatively accurate predictions based on conclusions derived from their claims.   Consequently, they are extremely useful in advancing our understanding of the world and our interaction with it, according to our aims and purposes.

Coherence

A theory in order to not be falsified must be internally and externally consistent.  We can think about this in terms of coherence.  First, the theory must be internally coherent: any claim that the theory makes must not contradict any other claims by the theory.  Such contradictions can be logical, or less strongly, physical.  Even better is the case when the claims are supportive of each other (without being simply alternative ways of saying the same thing).  Second, the theory must be externally coherent: it must not contradict (unless it is challenging the existing paradigm) any of the best scientific theories.

Informative and Explanatory

While there are perhaps other virtues that could be considered, let us consider a final one here.  We do not want theories that are merely predictive and accurate.  We want to understand why.  Thus, we expect a good scientific theory to be informative, to explain why things are the way they are in the world.  It will postulate the causal mechanisms that explain why something happens the way the theory accurately predicts.  It will provide direction for new avenues of research in light of those causal explanations.  In short, we do NOT want a black box, no matter how accurate that black box may be.

Data Science and Model Virtues

So how can the above be applied to data science models?

Falsifiable

A data science model must be falsifiable.  It must make predictions (i.e., hypotheses) that are capable of being false, and are tested accordingly.  This is why separating one's data into a training set, test set, (and verification set) is so important: it keeps one's model falsifiable.  When one builds a model on all of the data, one can have an extremely accurate model when only looking at the data at hand.  However, one is in danger of overfitting the model.  One is modeling aberrations, errors, outliers, or biases in the sample data, and consequently, the model will not generalize to future data.  It has NOT captured the real relationships underlying the data.  Using a hold out test set can keep your model honest, and make sure that your model will generalize to data that it has not seen before.

Furthermore, doing so prevents you from refitting the model with each new addition of data.  If one were to receive data on a daily basis, and on that basis, retrained the model, and if that model significantly changed each day, how confident would you be that your model was going to predict well?  If it would predict something today and something different tomorrow, then your model is not stable and it is not going to make accurate predictions.  It is no longer useful.  It is as though your model is changing its mind every day, changing with the wind, and never subjected to critical scrutiny because it is always explaining the latest data without being held accountable for the inaccurate predictions it is making.  This would be a pseudo-scientific model.

 

Elegance and Parsimony

A data science model must be as simple as possible or as is necessary, according to one's purposes. Why?  Again, it is more likely to be "true" in the sense that one is more likely to have captured that actual relationships among the independent variables and their relationship to the dependent variable. But there is a challenge here, because more simple data science models tend to not be as accurate or predictive, and this can be due to excluding variables that are predictive, even to a small degree.  So we don't want a model to be too simple, and yet, we don't want it to be too complex either given overfitting.  We want to have a model that is as simple as possible without sacrificing accuracy and one that generalizes well when tested. 

 

Predictive and "Accurate"

A data science model must be predictive and accurate.  This is the whole point!  We want to accurately predict unknown values.  If a model doesn't do this, it doesn't matter if it is elegant or falsifiable.  It isn't true.  It does not accurately model reality.  Your model must generalize to new data.

Coherence

A data science model must be coherent.  I suppose one could have a model that contains a variable that is nearly the opposite of a different variable in the model, and the model could use them both.  While possible, I am not sure that both variables would survive even minimal feature selection.  Nevertheless, if your data science model is incoherent in some way, correct it, or look into why your model is paradoxical in this way.

 

Informative and Explanatory

Does you data science model explain, inform, or illuminate the relationships among the variables you are using to predict?  That is, when you look at the coefficients for your linear model, or the branches in your decision tree, do you understand or get an "aha"?  A good model will help us understand what is really going on.  This is especially important when one wants to know what action to take.  Is it better to add square footage or add a new roof if one is trying to improve the resale value of a home?  A good model should be able to tell us and quantify an answer.  This is where avoiding overfitting is so important, because an overfitted model will not have stable or reliable relationships among its variables, and so these cannot be relied on for informing decisions.

There is a downside though, in that some types of modeling are extremely accurate (i.e., neural networks) but are very difficult to interpret.  This is not always a problem if one does not need to understand why the model predicted in the way it did.  If the outcome is all that matters, then interpretability is not as important. 

 

Conclusion

Unfortunately, there are no hard and fast rules here for guidance on how to create good models, which is why model development in data science can be likened to an art or skill.  But with practice, one can develop this skill.  Consider these high level principles when creating your model.  Reference them and work to make sure that either your models have these virtues or that you have really good reasons for lacking them.  If you do so, you will have a good model.

 

State and National Population and Voting Trends

The content of my final project for my Master of Data Analytics program class on extracting data, web scraping, data transformation, and data storage using R, MySQL, MongoDB, Neo4J, and other technologies is located here.

Due to formatting and file size issues, I have not posted the full text and images below.  However, here are some snippets to spark your interest in clicking on the above link to read the full text.

-------------------------------------------------------------------------------------------------------------------

Voting Trends

At the national level since 1789, the maximum electoral percentage for any candidate in a given voting year has jumped around wildly. But a very loose trend (as given by geom_smooth) would suggest that voters were more unified at the founding with a decline bottoming out right before the civil war. Electoral percentage for the winning candidate increase about 1950, when politics began to get more divisive again. This brings us to the most recent election, which was also very divisive.

Notice the only dot below 50% occurred in 1824 in a contest between Andrew Jackson and John Quincy Adams. While Jackson got more electoral votes, the election went to the House of Representatives for a vote because no candidate got a majority. Adams was then elected President by the House.


The popular vote percentage tells a similar story, but perhaps less extreme because it is the popular vote percentages as opposed to electoral vote percentages. We can see that the highest popular vote percentage was 61% in 1964 by Lyndon B. Johnson. The least was in 1860, when Abraham Lincoln received only 39.9% of the popular vote.




How does the national population relate to the national popular vote? As you can see by the graphs below, the relationship is very linear. This is not especially uprising, for as the population increases, so does the voting population. What is more revealing is that a look at the ratio of popular vote to population over time shows that this ratio is increasing. The trend is not perfect (people voted less prior to and during WWI), but overall we see an increase in voting rate that is slowing over time. In other words, more people are voting in each election, but that increase is getting relatively smaller and smaller. Perhaps the ratio will get closer to 45%, but never pass over.



What about candidates and parties? We can find the winning candidate and party for each state for each voting year from 1824 - 2016.  We can then visualize the information using the mapping capabilities from ggplot2. We produce a map of the United States for each voting year 1824 - 2016 and color each state with the winning party. The Democratic party is in blue while the Republican party is in red. Other parties are in various colors.

There is lots of interesting information here. We see in 1860 that the south all voted for “Southern Democratic” party. Abraham Lincoln won the election and we can see that in 1864, none of the South voted (this was during the Civil War). It’s also interesting to observe that the “Democratic” party used to be the conservative and state’s rights party of the south. From 1876 through about 1960, Texas and other deep south states were typically Democratic even when most of the other states were Republican. After 1960, the south transitioned to becoming consistently Republican by 1980. We can also see an emergence of the Democratic west coast, New England, and Great lakes states vs. a Republican midwest and south beginning in 1988. This is the pattern we have at present.



Before moving on, consider one last look at the map. In particular, compare 1900 to 2012 (see below map). Notice that they are almost exactly flipped: red states are now blue, and blue states are now red. Now compare 2012 to 2016, and notice that many New England and Great Lakes states have switched from blue to red. Could this be the start of another significant meaning change in the terms “Republican” and “Democrat”? We shall have to wait and see.

Please click here to view the full project text and analysis.