Classifying NFL Team Logos with Tensorflow Keras

Introduction

In fulfilling the final project requirements for a class on "Big Data and Machine Learning", which is part of my continued MS Data Science studies, the below report was created.  Some modifications have been made, but the substance remains largely the same.

The original report is here: https://github.com/anrcarson/CUNY-MSDA/blob/master/DATA622/finalproject-anrcarson/Report.md

The Task

Build an image recognition model and summarize your results. You may choose any data set that you are interested in exploring.

The Solution

Overview

For my project I downloaded images of NFL logos for the four following teams that make up the NFC West: Cardinals, Rams, Seahawks, and 49ers.  My task was then to build an image classifier that correctly classified each logo image to the corresponding team.  I followed TensorFlow Keras tutorials (relying most heavily on the link here: https://www.tensorflow.org/tutorials/images/classification), as well as code snippets from around the internet, to build an end-to-end solution.  The final code (imageClassification.ipynb) ran on my personal machine.

Here are the files/folders contained in my final GitHub repository (link is here: https://github.com/anrcarson/CUNY-MSDA/tree/master/DATA622/finalproject-anrcarson):

• ModelImages: the results of downloadpics.py (the code used to search for and download images).  Sorted into train/test sets.
• Screenshots: screenshots of my progress along the way.  Images referenced below.
• README.md: initial instructions as well as additional instructions for the project.
• Report.md: this file.  The final report.
• downloadpics.py: the code used to leverage the Bing Image Search API to find and download images of NFL team logos for use in building the model.
• imageClassification.ipynb: the image processing and model building, training, and testing code.  

Image Search

I used the Bing Image Search API, available via Azure Cognitive Services, to search for images relating to NFL team logos.  After signing up for Azure and setting up the API, I used the key provided along with sample code to do my automated searching. For example, one search was for "seahawks logo", just as you would do an image search in Bing using a browser.  The results were then paged through, while each main image url was used for downloading the image locally for later use in training the model.

Not all images were useful or accurate for my purposes.  I manually combed through the roughly 800-900 images returned for each of the four teams to only retain those that I judged to accurately match the team in question.  I also needed to remove animated GIFs that created errors in code processing to keep only JPEGs.  This left about 600-700 images per team for a total of 2653 images.

Split into Train and Validation

Having gotten the images, I needed to split them into a training set and validation set for use in training the model.  I created some code to do this automatically using a 67% training 33% validation split.  The code pulled evenly from each team class.  This gave me 1765 training images and 888 validation images.  In exploring documentation for something else I learned that I could have used the "validation_split" in the image_generator to do this for me, but as this was no longer needed, I did not redo this (although I will redo this in the future if I decide to extend this project).

Training

I did several rounds of training using different methods for preventing overfitting, adding additional layers in the model, and using different kinds of image augmentation.  In particular, I experimented with the following:

• With and without data augmentation: image rotation, width shift, height shift, horizontal flip, and zoom. (see "trainingImages" and "trainingImages_withDataAug" screenshots for difference below).
• Without augmentation:
• 
•  With augmentation:
• 
• With and without dropout layers
• With and without learning rate schedules
• With and without early stopping
• With and without L2 regularization

The basic model was a convolutional neural network (CNN).  Using the example of the tensorflow tutorial, I used a sequential model with three Conv2D layer / MaxPooling2D layer combinations (using ReLu activation) before a flatten/dense layer, with a final dense layer of 4 softmax outputs.

Results

A brief history of the results is as follows:

• The initial model with 15 epochs had a training accuracy of 0.99 with a validation accuracy of 0.89.  Not bad, but quite a gap between training and validation.(results_01_originalCNN)

• Adding dropouts lowered the training accuracy to 0.92 and increased the validation accuracy to 0.85.  While decreasing the accuracy this did bring the training closer to the testing accuracy, avoiding overfitting to some degree. (results_02_addDropouts)

• Increasing the epochs to 25 increased the training accuracy back up to 0.99 and the validation accuracy up to 0.89.  No real improvement except that having more epochs gave the model more time to learn. (Note: I think this run may have started on an already trained model, hence the immediate high level of accuracy.) (results_03_25Epochs)

• Adding all of the above data augmentation with 15 epochs lowered the training accuracy to 0.69 and the validation accuracy to 0.69.  This brought training and validation accuracy into alignment. (results_04_15Epochs_DataAug)

• I removed the horizontal flip as I didn't think this was fair to the logos (is the 49ers logo correct if it is backwards?) and increased to 25 epochs while implementing early stopping.  Training accuracy improved to 0.74 and validation accuracy to 0.81 (Note: I did not have data augmentation on the validation data, so these were easier to predict).  So a longer training time was important to improving the accuracy. (results_04_25Epochs_RemoveFlip)

• Adding a learning schedule and L2 regularization in one place increased the accuracy to 0.80 and the validation accuracy to 0.84.  These adjustments appeared to help the model learn better.  However, the model did not have to use early stopping, meaning that even 25 epochs was not long enough for training. (results_05_25Epochs_L2_LearningSch)

• I increased to 100 epochs.  The training accuracy went to 0.87 and the validation accuracy went to 0.88.  The model early stopped at 57 epochs.  I saved this model as the "data augmented" version. (Note: I think this run may have also started on an already trained model). (results_06_100Epochs)

• I removed the data augmentation and reran the training model with the other additions in place.  Stopping at 52 epochs, the model had an accuracy of 0.98 and a validation accuracy of 0.90.  I saved this as the "no data augmentation" version of the model. (results_07_100Epochs_NoDataAug)

Prediction and Misclassified Images

To view the misclassified images, it was necessary (in so far as I could find based on documentation and online discussion) to predict the training images with the shuffle flag set to False.  I then converted the probabilities output by the model into the appropriate class value with the highest probability.  Then I matched these with the correct labels. The model with data augmentation using this method predicted 88.7% correctly on the training data (accuracy_modelWithDataAug).

The accuracy when using the "no data augmentation" version of the model and this method was 99.6% on the training images (accuracy_modelWithNoDataAug), meaning that 7 were misclassified.  I viewed these individually.

Six of the seven images (see misclassifiedImages below) were actually Rams images, and one was a 49ers image.

Here is my best guess as to what happened.

• 49ers shirt classified as "Cardinals"/ Rams shirt classified as "Cardinals": if I look at the training data, the Cardinals have about 20 images of shirts with the Cardinal's logo on it.   The 49ers had fewer shirts and these didn't look as similar to the misclassified image (viewed straight on and flat image).  The Rams had only about 5 shirts.  Hence, the model learned that if it is a certain kind of shirt, it is "Cardinals".
• The exception to the above was a Rams shirt that was labelled as "FortyNiners".  This shirt is 3D looking and viewed from an angle.  The 49ers have a lot of these similar images whereas the Rams have very few.
• The Rams skull is labeled as "FortyNiners": The 49ers have three similar skulls (and another less similar one) in the training data while the Rams have one.  So "if it's a skull, its 'FortyNiners'" was the learned rule.
• Rams carpet circle labeled as Seahawks: Not really sure why.  There were other Rams carpet circles that correctly classified.  The Seahawks logo does have similar colors and geometry, and there were lots of round Seahawks logo images in the training set, so perhaps this is why.
• Rams lettering labeled as "FortyNiners":  the colors here are gray and white.  The 49ers do have some images that are similar, but perhaps it is the fact that the 49ers logo is usually an "SF" or says "49ers".  Meanwhile, the Rams logo usually is just the Ram, the horns, or the the Ram along with the spelled out "Rams".  But not really clear why this was misclassified.
• Rams player labeled as "FortyNiners": The 49ers have a lot more images of players than the Rams do.  I think absent a clear logo or symbol (e.g., side view of the helmet), the learned rule was "if a player, it is FortyNiners".

Future Enhancements

There are lots of variables I could still tweak in the model.  I largely left the initial variables in place that the tutorials used because they seemed to be working well in my testing, they presumably worked well for the tutorials, and they seemed to use best practices from the literature I have read for this sort of problem.  In the future I would experiment with different cost or activation functions, adding additional layers, changing the number of nodes within layers, and changing image sizes or batch sizes.  The changes and combinations I could make to the model are virtually infinite, and while I would expect some improvement, I don't know if much would change in terms of accuracy and generalizability with these model-specific tweaks.

However, I think the biggest improvements would come from better data.  I would try to get more data for each of the teams, and I would try to make the images more balanced across teams.  For example, I could try to balance out the number of images of players per team.  I could remove the "skull" images from the data unless there were similar images available for all of the teams.  That is, I would try to make it so that the only real difference among the images considered as a set for each team is the identifiable logo, symbol, etc. for that team.  I would do more experimentation with the data augmentation controls as well in order to make the model more generalizable to new images.  And I would experiment with the automatic splitting of the data into training and validation sets in hopes that the model would become more generalizable (and not just learn what was in a static training set).

Another future enhancement would be to expand the model to include other teams, and eventually all 32 NFL teams.  I would want to get the model performing very solidly on the current scope of teams and would definitely experiment with the tweaks and data augmentation mentioned above before.  But it would be nice to have a model that covers all teams.  This would be a much more extensive data collection effort, but could be a fun continuation of this project in the future.

Conclusion

My goal in this project (apart from satisfying class requirements) was to find a dataset that I took to be interesting and to build an image classification model on it using the latest deep learning packages and techniques.  By gathering data that was "out in the wild" and not prepackaged, gathered, or preprocessed, I am sure that I could use this approach for other image sets that are also not nicely packaged for image classification (e.g., work related images, personal images).   By leveraging TensorFlow Keras, I have utilized one of (if not the) most popular deep learning code package in current use.  And by following TensorFlow tutorials with not an insignificant number of enhancements and changes, I have utilized the most current methods for creating and fitting an image classification model.  In short, I have built a relatively simple but modern end-to-end image classification deep learning model that can be used as a template or extended for additional projects.

Text Mining the Quran

Note: For the original jupyter notebook file, code, and data upon which this post is based, please visit my GitHub.

Introduction

I am interested in learning more about what Islam teaches and what exactly is in the text of the Quran(/Koran/Qu'ran), especially because of its current political relevance both in the United States and abroad. Consequently, to begin to do this, I will focus on text mining the text of the Quran. I intend to scrape the entire text from available sources online, grouping the text by chapter and verse. For each chapter, I will find the top words, keywords, and sentiment. I will also find meaningful clusters of words (i.e., bigrams and trigrams). These analyses will be visualized, along with any trends or patterns I find. Finally, I will summarize my conclusions.

I should note that my intentions in performing this analysis are purely descriptive with some inferences as to the meaning of my findings. I make no judgments on the truth of what the Quran contains or the correctness of its teachings, and nothing below should be construed as such. Also, as I am not a Quranic scholar, and as there are different traditions and methods of interpretation of the Quran, my conclusions below should be considered as an exploratory starting point in understanding the Quran, and not as representing any authoritative interpretation or conclusions.

The Quran

Before beginning, a quick introduction to the nature and structure of the Quran may be helpful. The Quran is the primary religious text used by Muslims, similar to the Bible (especially New Testament) for Christians and the Torah (the first five books of the Old Testament) for Jews. It contains revelations and instructions from God (in Arabic, "Allah") given to Muhammad, the prophet of Islam, from between 610 AD through 632 AD when Muhammed died (reference: https://en.wikipedia.org/wiki/Islam). The original and authoritative text is in Arabic, although there are translations into other languages as well (e.g., English). However, these would not be considered authoritative, and, as is a danger in all translations, may not convey the full and intended meaning due to cultural, textual, and historical differences. The Quran contains 114 Chapters ("Surahs") and is organized roughly according to length, from the longest chapter to the shortest chapter. However, the chronology of the chapters is also known, so one can also analyze chapters in their historical order, which I will do so below.

Packages

The below modules include the packages I will be using for this project. In particular, I will use:

• Beautiful Soup - text extraction
• re - regular expressions
• numpy - arrays
• pandas - data storage and transformation
• matplotlib - data visualization

Get Text

The below code blocks get the text from the chosen website (http://www.clearquran.com). The first three define functions that:

• get the website links
• get the website HTML
• get the text from the website HTML

Note: I chose the website above because it was one of the first returned in a search and because the structure of the website provided a relatively easy way to scrape and parse the data. I make no claim that this translation is better or more authentic than other English translations. Consequently, any conclusions I derive from the text depend on the accuracy of the textual translation itself.

The fourth code block uses the three functions above to get the website link for each chapter ("Surah") of the Quran, get the HTML for the website, get the text for each website (i.e., the verses for the chapter), clean the text (e.g., remove punctuation, numbers), split the words, do additional clean up, and then assign the chapter number, verse number, and words to a dataframe. The final result is a dataframe of Chapter, Verse, and Word for all of the words in the Quran.

The data is then saved to a CSV file for future use.

Clean Up

Before analysis, we need to clean up the data. I add column names to the dataframe and remove stopwords. Before removal, there are 146,279 words. After removal, there are 63,145 words.

Analysis

Word Counts

Now we can analyze the data. First, we can see that, indeed, the chapters are roughly organized from longest to shortest by looking at word counts for each chapter. There is a short introductory chapter, but this is followed by the longest chapter. Apart from occasional dips downward and then back up, the overall trend is from longer chapters to shorter chapters.

How does this compare with the Chronological Ordering of the chapters? Below, we first read in the chronological ordering from a website. Then, we compare the Chapter order with the Chronological order using the number of verses. As you can see, they are very different.

The chronological order begins dipping right at the beginning, but then climbs to an average maximum about 50 chapters in. This decreases on average until the upper 80s when there are several very large spikes, including the longest chapter of the Quran, before continuing a downward trend. There is a spike in long verses at the end as well. It is reasonable to conclude that these spikes relate to important periods in Muhammad's life when longer revelations would be necessary. For example, it would make sense to have longer revelations prior to Muhammad's death to guide the people after he had died.

The spike in the upper 80s appears to be related to the location of the revelation. The spike occurs at chronological order 87 (marked by a green vertical line), and corresponds to the first revelation in Medina following Muhammad's leaving Mecca. Such a lengthy revelation would be necessary to guide the community of believers in a new way of life in a new town, away from the persecution faced in Mecca.

Term Frequency

On to the actual text. Which words occur the most? When we look at term frequency over the whole of the Quran, we see that the word "God" ("Allah") occurs the most, over 2500 times. "Lord" is second with just under 1000. We also see "people", "believe", "earth", "merciful", "messenger", "punishment", "heavens", and "truth".

From this, we can infer that the Quran is foremost about God, whose most commonly ascribed attribute is mercy. It is addressed to a people or people in general, calling them to believe in God's truth, which is conveyed via a messenger (i.e., Muhammad). For those who believe, heaven awaits. For those who do not, punishment.

Normalized Term Frequency

Suppose we look at individual chapters. To compare across chapters, we should normalize the word counts within a chapter by dividing a word's total counts in a chapter by the chapter's total word count. The below plot shows the top 30 word-ratios, along with the associated word and chapter. We can see that "God" appears quite a bit. Other interesting words include "worship", "serve", "evil", "deny", "marvels", "condemned", and "shocker".

TFIDF

What we are really trying to get at are keywords. A useful way of getting at keywords is to use TFIDF, that is Term Frequency (and) Inverse Document Frequency. The Term Frequency, which we already have, is the number of times a word appears within a document. Since some words occur quite a bit but are not particularly important in determining what a document is about (e.g., "said"), it is useful to prioritize unique words within the collection of documents as indicative of what each document contains. This is done through the IDF. However, since many rare words may not really be that important, one can combine TF andf IDF to score words according to how often they occur and their uniqueness. Thus, one can pick out words that truly are key for any particular document.

However, when we look at the top words by Chapter and TFIDF, we do not have any new discoveries. "God" is still the most important keyword. We also see "marvels" again, along with "said" and "say".

    WordUpper  Chapter       TFIDF
148       GOD      2.0  249.000000
48        GOD      4.0  212.000000
84        GOD      3.0  189.000000
44        GOD      9.0  156.000000
15    MARVELS     55.0  143.765171
92        GOD      5.0  137.000000
128      SAID     12.0  120.274019
971      SAID      2.0  117.023370
577      SAID      7.0  113.772721
784    JOSEPH     12.0  111.302068
448       SAY      6.0   99.788023
619    PEOPLE      7.0   93.973035
23        GOD     33.0   84.000000
368       GOD      6.0   83.000000
13      LORDS     55.0   81.999836

What if we normalize TFIDF? That is, we use the normalized word counts for TF in the TFIDF equation. Does this bring new insight? Yes. We see lots of new words in the top 15: "loser", "hates", "crusher", "plenty", "sacrifice", and "encourage". It is important to note that many of these come from the same chapters, 108 and 109. It would appear that these chapters have very strong keywords in them compared to the rest of the Quran's chapters.

    WordUpper  Chapter  NormalizedTFIDF
2       SERVE    109.0       736.307505
21    SHOCKER    101.0       627.502961
33      LOSER    108.0       582.681321
31      HATES    108.0       582.681321
15    MARVELS     55.0       570.294902
20  CONDEMNED    111.0       562.494348
70     BEGETS    112.0       509.846156
82    CRUSHER    104.0       502.002369
4     MANKIND    114.0       490.658890
37     PLENTY    108.0       471.084614
35  SACRIFICE    108.0       471.084614
1     WORSHIP    109.0       458.559404
5    SECURITY    106.0       451.093997
3        EVIL    113.0       436.671046
60  ENCOURAGE    103.0       425.495780

How often is a word the most important keyword in a chapter in the Quran? We can find the maximum normalized TFIDF word for each chapter, and then by grouping on each distinct word, we can get a count for each keyword. In the plot below, we see that "God" is the most important keyword in 32 of the 114 chapters.

There are 73 distinct words that the most important keyword in a chapter. Most only occur once, in fact, only the top 5 shown below are the most important keyword in a chapter more than once. The only word we haven't seen yet is "ease".

 

Bigrams

Let's move on from single words to bi-grams, that is, a pair of consecutive words. From these we can get a better sense of the context or meaning of words, since we know what other words surround them. In the below code, I find the bi-grams after having removed the stop words. The top 30 most commonly occurring bi-grams are then plotted.

How might we interpret the bigrams below? Looking at the plot, we see the most common bi-gram is "heavens earth". We have seen both words previously. This may refer to "[the] heavens [and the] earth", which could be a way of expressing God's power and dominion over both "the heavens and the earth". "Everything [in the] heavens" "belongs [to] God", who is the "Lord [of the] worlds".

We also see "God gracious", "gracious merciful", and "forgiving merciful", suggesting that God is gracious and merciful and forgiving. Another aspect of God is that "God knows", perhaps referring to God's omniscience or knowledge of all things.

Finally, we see "day [of] resurrection", referring to end times and one's eternal status. For those who "turn away" and do not "believe God" or worship gods "besides God" there is "painful punishment" awaiting. Thus, one should "fear God" and "know God" and listen to "God['s] messenger" and the "good news". Such news may be especially directed at the "children [of] Israel", that is, the Jewish people.

Trigrams

Now let's look at trigrams, that is, consecutive groupings of three non-stopwords. Using the same process as above, we can see similar groupings of words and infer a similar overall message.

For example, we see "God [is] gracious [and] merciful", "[in the] name [of] God [the] gracious", and "God [is] forgiving [and] merciful". "God [is all] hearing [and all] knowing", "God [is] mighty [and] wise", and God "knows best". We may infer that God "created [the] heavens [and] earth" and "everything [in the] heavens [and] earth " belongs to God. We are called to "believe [and to do] good deeds" or "righteous deeds". We must "fear God [and] obey", and as a "people [we should] worship God." We will face "God [on the] last day" to be assigned a place of "abiding" "forever". Those that the "Lord[']s marvels deny" should heed these warnings.

Something new and very interesting is "Jesus son [of] Mary". This refers to Jesus, who instead of being considered divine as Christian's do, is one of the prophets of Islam but is not considered divine.

Sentiment Analysis

A common textual analysis is to look at sentiment. So we can ask, what is the sentiment over the course of the Quran? We can look at this by actual chapter ordering, as well as chronologically. I adapt from an approach located here: https://mran.microsoft.com/posts/twitter.html. In essence, for each chapter, I do a count of the positive words and a count of the negative words, find the difference, and return that difference as the sentiment score. While such an approach has obvious problems (e.g., "not funny" is counted as positive since "not" is ignored and "funny" is a positive word on its own), it is a useful starting point.

The list of positive and negative words I used can be found here: http://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar.

Now we are ready for analysis. First, what does the sentiment look like by traditional chapter order? We can see that there are wide swings in sentiment, but these stabilize as time goes on. Why? After some exploration, we can see that, since the Quran is roughly ordered by chapter length, there are more opportunities for positive or negative words to occur, and this makes the longer chapters more likely to have higher or lower sentiment.

To adjust for this, we can normalize the sentiment by chapter length, that is, by calculating the average sentiment per verse in the chapter. When we do so, the sentiment looks a lot more balanced. There are some low points in the middle and some high points at the end, but overall, the sentiment appears to go back and forth.

We can do the same thing chronologically. In the absence of normalization, it would appear that until about the 40th chronological chapter, the sentiment is fairly stable and mostly positive. About chronological chapter 40, the sentiment swings back and forth wildly before settling out again. This stability ceases right after the change from Mecca to Medina (the red vertical line) in which the sentiment swings to positive and negative extremes again. The lowest sentiment in the whole Quran appears near the very end in chronological chapter 112 (traditional chapter 5).

When normalized, the first part of the Quran in chronological order has the strongest sentiment, and it is mostly positive. The middle stabilizes and is almost flat until the transition from Mecca to Medina, when the strong sentiment re-emerges. While it is mostly positive, the lowest normalized sentiments occur in chronological chapters 104 and 105 (traditional chapters 63 and 58).

Which words are in the chapters that have the strongest sentiment, both positive and negative? We can see which chapters have the highest and lowest sentiment and normalized sentiment in the tables below.

On the positive side, traditional chapter 28 is a longer chapter and has words like "given", "land", "guidance", "know", and "resurrection". Traditional chapter 103 is very short and has the words "good", "gracious", "merciful", "patience". Traditional chapter 110 is also short, and has words like "praise", "merciful", "celebrate", and "victory".

On the negative side, traditional chapter 5 is a longer chapter and has words like "fear", "disbelieve", "punishment", "hostile", and "hunger". Traditional chapter 58 has words like "punishment", "Satan", "oppose", "estrange", and "lie". Traditional chapter 63 has words like "hypocrites", "repel", "deluded", and "evil".

Obviously, to really understand what these chapters are about and why they have very positive or very negative sentiment, one must read them to understand the context of the positive/negative words that determine the sentiment.

 

Conclusion

So what is the Quran really about? While the above analysis can't provide any nuanced or really specific conclusions, we can say that the Quran (and Islam) is generally about:

• God, who is gracious, merciful, forgiving, powerful, knowledgeable, and creator of the heavens and the earth
• God's instructions on how to live one's life in proper obedience and worship to God
• The eternal reward or punishment due to humans depending on how they live their lives in obedience or disobedience to God

In this, we can say that the Quran (and Islam) is in very general agreement with the other Abrahamic faiths of Judaism and Christianity in these general teachings of the religion. That is, at a very general level, these three faiths appear to be in agreement on the general nature of God and the purpose and destiny of humanity. However, a more detailed comparison of these religions would likely reveal great and important differences. For example, on the question of "who is Jesus?", each gives a very different and important answer.

While such differences should not be minimized or ignored, nor should they detract from the commonalities and agreement that these faiths have with each other. Hopefully, a greater understanding of these similarities and differences can lead to common cause and coordination in confronting the challenges the world faces today. My hope is that the above analysis has provided at least a small and initial step forward in that greater understanding, not just for me, but for you as well.

Text Mining the Bible

Introduction

What is the Bible about?  There are many ways one could answer this question, and even more ways one could go about finding an answer to the question.  One could read the Bible or read books about the Bible.  One could look at the scriptural text, the theological formulations, the historical narrative, or adopt another lens by which to make sense of what the book as a whole is all about.

In what follows, I offer a data science approach to begin to answer this question through the use of keyword relevance analysis and data visualization.  Such an analysis can provide quick insights into the key words and themes that can then be understood visually.  While extremely simplistic and nowhere near as rich, complete, or meaningful as actually reading the Bible or reading books on the Bible, such an analysis can provide guidance as to what is important.  I detail the method and show the results of this analysis in what follows.

The Method

I wrote a Python script that scraped the Biblical text book by book, chapter by chapter from Biblegateway.com.  I selected the NRSV translation and used the Catholic selection and ordering of books in the Bible.  Once the book text was parsed and cleaned (ok, not perfectly, but good enough for demonstration purposes), the whole book was submitted to the Alchemy API for keyword relevance analysis.  The output of the API was a keyword (or key phrase) (e.g., "God") with the relevance or importance of the keyword within the context of the book (e.g., 0.95).  The higher the relevance, the more important the keyword was.  After compiling the results, I had a list of about 50 keywords by relevance for each book of the Bible.

I took the results and brought them into Tableau for interactive filtering and visualization.  Screenshots below of the results are taken from the resulting workbook.

Here are some links for those who may be interested regarding the nuts and bolts of the analysis:

• BibleGateway:
• source for text
• GitHub
• The code for web scraping and submitting to Alchemy API.
• You will need your own API key.  Modify the code as necessary to suit your needs.
• Documentation for Alchemy API
• Tableau workbook

Here are some high level results of the analysis with some simple conclusions.

Results: Most Import Keywords

So what is the Bible about?  From the above method, we can look at keywords and phrases by total counts and relevance.  We'll look at these in turn.

Total Counts

What are the most common keywords?  With 73 books in this Biblical list, a keyword could have a total count of 73 if it was a keyword in every book of the Bible.  According to the Alchemy API, the top 10 keywords by total count in the Bible are:

1. God (48)
2. Lord (46)
3. people (44)
4. house (26)
5. things (22)
6. Lord God (22)
7. Israel (22)
8. son (21)
9. land (20)
10. king (19)

Not surprisingly, the Bible is about "God" or the "Lord".  It is also about a "people", namely, the Israelites, and the story of their establishment in the "land" of Israel.  Much of the Old Testament talks about the various rising and falling of "king"s and the establishment of the "house" of the Lord (i.e., the temple) or being a member or descendant of a "house" (e.g., the house of David).   As lineage is important, "son" is used a lot to explain ancestry, and in the New Testament, this word takes on a special meaning as part of the phrase "son of God". 

Here are the top 3 keywords (God, Lord, people) over the range of the Bible.  While not categorized as relevant in every book of the Bible, "God"/"Lord" is very relevant throughout most of the Bible.  The word "people", though less relevant on average, is also found throughout the Bible.  This would suggest that the Bible is fundamentally about "God" and the relationship of "God" to a "people".

Noticeably absent from this list, is "Jesus", whose top entry is "Lord Jesus Christ" tied at #12 with a total count of 15.

Maximum Relevance

What are the most relevant keywords?  A keyword may only occur a few times but be extremely relevant in context, while another word may occur more often but may not be relevant at all.  So while total count does give an indication of relevance, it is not exhaustive.  According to this Quora post, the Alchemy API calculates relevance by using word position, context of other words, how many times it is used, and other statistics.  However, the specific details are not documented anywhere that I could find.

What are the most relevant keywords/phrases?  I looked at the maximum relevance for a keyword in the whole Bible.  In order of most relevant, the top 10 keywords/phrases are:

1. Lord Your God (0.998)
2. God (0.996)
3. beloved speaks (0.992)
4. son (0.990)
5. savior Jesus Christ (0.989)
6. shall (0.989)
7. Lord (0.988)
8. Judas (0.988)
9. Jesus Christ (0.984)
10. Holy Spirit (0.984)

Again, we can infer that the Bible is about "God", or more specifically (as addressed to the Israelites) the "Lord Your God".  "Beloved speaks" is interesting in that it comes from the Song of Solomon, in which Solomon writes poetry to his "beloved".  We also see again the importance of "son".  But now we also see "Jesus Christ" enter into the picture as important, as is the "Holy Spirit".  Thus, each person of the trinity is considered relevant.

The occurrence of "Judas" is also very interesting.  However, it is not a reference to the Judas who betrayed Jesus, but a reference to "Judas Maccabeus" from the book of 1 Maccabees.  The word "shall" comes from the book of Micah, who is a prophet and speaks of many things that "shall" happen.

Here are the top 5 (Lord Your God, God, beloved speaks, son, savior Jesus Christ).  We have seen that "God" is used throughout the Bible.  "Lord Your God" is used primarily in the early part of the Old Testament and then again in the prophets.  The word "son" is used heavily in the historical books of the Old Testament, and then again in the 4 gospels: Matthew, Mark, Luke, and John.  "Savior Jesus Christ" only makes one appearance in 2 Peter.

Total Counts and Maximum Relevance

We have seen that some words have high counts but lack relevance, while others have relevance but lack high counts.  Which keywords have both?  I increased the Total Counts filter and the Max Relevance filter until I had 10 keywords.  Choosing different thresholds would result in a slightly different combination, but my selection requires each keyword to have a max relevance above 0.90 and a Total Count greater than 12.    This results in the following list:

• Father
• God
• Holy Spirit
• Israel
• Jesus Christ
• King
• Lord
• Lord God
• Lord Your God
• Son

We have seen most of these already.  The exception is "Father".  It is a keyword 13 times and has a max relevance of 0.933.  It occurs throughout the Bible, but reaches a high point in Matthew.   Matthew opens up with a genealogy of Jesus in the form of "X was the father of Y, and Y the father of Z, and Z...", and this goes on for many lines. 

Most Relevant Keyword In Book Total Counts

Which words are consistently the most relevant in a book?  If we find the most relevant word in a book and then find the total counts of these words, which ones are on top?  The top 8 (those occurring more than once) are:

1. Lord (17)
2. God (14)
3. Son (4)
4. Christ (4)
5. King (4)
6. Christ Jesus (3)
7. Jesus Christ (3)
8. Jesus (2)

Again, "Lord" and "God" are consistently the most important.  We have also seen "son" and "king" before, as well as "Jesus".  Of note is that "Christ" now appears on its own in addition to being attached to Jesus as we have seen before.  The word "Christ" means "Messiah" who is the awaited savior and king of the Jews.  So there is the notion that Jesus is the "king" and savior of the Jewish people.

Keyword Category Total Counts

Many of these keywords could be further categorized or similarly grouped.  For example, "Jesus Christ" and "Christ Jesus" could be grouped with "Jesus".  After performing such groupings, what are the total counts now?  The top 10 are:

1. God (228)
2. Lord (150)
3. Family (149)
4. People/Israel (146)
5. Jesus (117)
6. Virtue (102)
7. Covenant/Law (61)
8. Body Part (58)
9. Land/Earth (56)
10. Humans (51)

"God" and "Lord" are still at the top, but next we have familial terms like "father", "son", "brother", "sister", and "child".  Then we have terms related to "Israel" or "People".  Next, any terms related to "Jesus".  After that, terms of virtue: peace, love, mercy, faith, hope.  Covenant/Law refers to terms related to the Old Testament law: priests, offering, covenant, and law.  Body Part refers to a mention of a body part like hands, heart, and eyes.  Land/Earth refers to the land, Earth, world, hill country, or some other generic description of location.  Lastly, humans refers to any non-familial term for humans: woman, man, and young men.

So we can see that this grouping still characterizes the Bible as about God, but perhaps it is about a relationship to people spoken of in largely familial concepts.  At the very least, family relationships are important and spoken of often.  Jesus is important too (more on that below).  A life of virtue in reference to (or perhaps in contrast with) the law/covenant of Israel is how one ought to live.  Such a vision has a broad scope, applying to the land and the Earth and humans of every kind on the Earth.

Specific Comparisons

Other specific comparisons could made of keywords and categories, but here are two interesting ones I saw.

Trinity: God/Lord, Jesus, Spirit/Holy Spirit

As we know, "God"/"Lord" is important throughout the Bible.  Once entering the New Testament, "Jesus" becomes equally important.  "Holy Spirit" has lesser importance, but is still prominent in the New Testament.  It is highest in the book of Acts, when it is more relevant than either "Jesus" or "God".

Virtue, Covenant/Law, Sin

The categories of Covenant/Law and Virtue seem to trade off in relevance.  In the beginning of the Bible, Covenant/Law keywords are most relevant.  In the prophetic books, Virtue keywords are more relevant.  In the beginning of the New Testament, both are important, but Covenant/Law is emphasized over Virtue. For the rest of the New Testament, however, Virtue is most important.  "Sin" appears here and there, but is only ever about as relevant as Virtue.

Other Insights?

While I could go on and on with more insights, I will leave that to you!  I have embedded the Tableau dashboard below.  You can also go here to view and use the dashboard.

Conclusion

So what is the Bible about?  From this simplistic keyword and relevance analysis, we can sum up the above findings and say that the Bible is broadly about:

• God (the Lord)
• The history of God's interaction with the people of Israel through the use of law and a covenant.
• The coming of Jesus, who is considered to be the Christ and king of the Jewish people, followed by the coming of the Holy Spirit.
• A shift towards a life focused on living by virtue instead of living (merely) by the law.

Much more can obviously be said here, but I think the main goal has been achieved.  That is, I have shown that a keyword and relevance analysis using data science methods (e.g., web scraping, APIs, data visualization) can reveal much about the important themes in a text, in particular, the Bible.

Buy or Sell? Stock Market Daily Tracker

With my life getting busier and busier, adding another item to the daily morning routine, no matter how short in the amount of time it takes, is to be avoided.  So as I have been observing the recent increases in the stock market and deciding that I should be paying attention more, I have been pondering about how I might do this in a more automated fashion.

Consequently, I decided to write a simple Python program that pulls some daily stock market information (e.g., today's opening, yearly high, yearly low), runs some calculations (e.g., opening / yearly high), and then after some simple conditional logic, tells me whether I should buy. This code runs on a daily schedule and opens up in a pop-up window automatically.

It is rather simplistic, but a quick glance at this every day is a simple way to keep an eye on the market and to be informed of action I should take in response.

The code is posted to GitHub if you are interested.