Text Mining the Quran

Note: For the original jupyter notebook file, code, and data upon which this post is based, please visit my GitHub.

Introduction

I am interested in learning more about what Islam teaches and what exactly is in the text of the Quran(/Koran/Qu'ran), especially because of its current political relevance both in the United States and abroad. Consequently, to begin to do this, I will focus on text mining the text of the Quran. I intend to scrape the entire text from available sources online, grouping the text by chapter and verse. For each chapter, I will find the top words, keywords, and sentiment. I will also find meaningful clusters of words (i.e., bigrams and trigrams). These analyses will be visualized, along with any trends or patterns I find. Finally, I will summarize my conclusions.

I should note that my intentions in performing this analysis are purely descriptive with some inferences as to the meaning of my findings. I make no judgments on the truth of what the Quran contains or the correctness of its teachings, and nothing below should be construed as such. Also, as I am not a Quranic scholar, and as there are different traditions and methods of interpretation of the Quran, my conclusions below should be considered as an exploratory starting point in understanding the Quran, and not as representing any authoritative interpretation or conclusions.

The Quran

Before beginning, a quick introduction to the nature and structure of the Quran may be helpful. The Quran is the primary religious text used by Muslims, similar to the Bible (especially New Testament) for Christians and the Torah (the first five books of the Old Testament) for Jews. It contains revelations and instructions from God (in Arabic, "Allah") given to Muhammad, the prophet of Islam, from between 610 AD through 632 AD when Muhammed died (reference: https://en.wikipedia.org/wiki/Islam). The original and authoritative text is in Arabic, although there are translations into other languages as well (e.g., English). However, these would not be considered authoritative, and, as is a danger in all translations, may not convey the full and intended meaning due to cultural, textual, and historical differences. The Quran contains 114 Chapters ("Surahs") and is organized roughly according to length, from the longest chapter to the shortest chapter. However, the chronology of the chapters is also known, so one can also analyze chapters in their historical order, which I will do so below.

Packages

The below modules include the packages I will be using for this project. In particular, I will use:

• Beautiful Soup - text extraction
• re - regular expressions
• numpy - arrays
• pandas - data storage and transformation
• matplotlib - data visualization

Get Text

The below code blocks get the text from the chosen website (http://www.clearquran.com). The first three define functions that:

• get the website links
• get the website HTML
• get the text from the website HTML

Note: I chose the website above because it was one of the first returned in a search and because the structure of the website provided a relatively easy way to scrape and parse the data. I make no claim that this translation is better or more authentic than other English translations. Consequently, any conclusions I derive from the text depend on the accuracy of the textual translation itself.

The fourth code block uses the three functions above to get the website link for each chapter ("Surah") of the Quran, get the HTML for the website, get the text for each website (i.e., the verses for the chapter), clean the text (e.g., remove punctuation, numbers), split the words, do additional clean up, and then assign the chapter number, verse number, and words to a dataframe. The final result is a dataframe of Chapter, Verse, and Word for all of the words in the Quran.

The data is then saved to a CSV file for future use.

Clean Up

Before analysis, we need to clean up the data. I add column names to the dataframe and remove stopwords. Before removal, there are 146,279 words. After removal, there are 63,145 words.

Analysis

Word Counts

Now we can analyze the data. First, we can see that, indeed, the chapters are roughly organized from longest to shortest by looking at word counts for each chapter. There is a short introductory chapter, but this is followed by the longest chapter. Apart from occasional dips downward and then back up, the overall trend is from longer chapters to shorter chapters.

How does this compare with the Chronological Ordering of the chapters? Below, we first read in the chronological ordering from a website. Then, we compare the Chapter order with the Chronological order using the number of verses. As you can see, they are very different.

The chronological order begins dipping right at the beginning, but then climbs to an average maximum about 50 chapters in. This decreases on average until the upper 80s when there are several very large spikes, including the longest chapter of the Quran, before continuing a downward trend. There is a spike in long verses at the end as well. It is reasonable to conclude that these spikes relate to important periods in Muhammad's life when longer revelations would be necessary. For example, it would make sense to have longer revelations prior to Muhammad's death to guide the people after he had died.

The spike in the upper 80s appears to be related to the location of the revelation. The spike occurs at chronological order 87 (marked by a green vertical line), and corresponds to the first revelation in Medina following Muhammad's leaving Mecca. Such a lengthy revelation would be necessary to guide the community of believers in a new way of life in a new town, away from the persecution faced in Mecca.

Term Frequency

On to the actual text. Which words occur the most? When we look at term frequency over the whole of the Quran, we see that the word "God" ("Allah") occurs the most, over 2500 times. "Lord" is second with just under 1000. We also see "people", "believe", "earth", "merciful", "messenger", "punishment", "heavens", and "truth".

From this, we can infer that the Quran is foremost about God, whose most commonly ascribed attribute is mercy. It is addressed to a people or people in general, calling them to believe in God's truth, which is conveyed via a messenger (i.e., Muhammad). For those who believe, heaven awaits. For those who do not, punishment.

Normalized Term Frequency

Suppose we look at individual chapters. To compare across chapters, we should normalize the word counts within a chapter by dividing a word's total counts in a chapter by the chapter's total word count. The below plot shows the top 30 word-ratios, along with the associated word and chapter. We can see that "God" appears quite a bit. Other interesting words include "worship", "serve", "evil", "deny", "marvels", "condemned", and "shocker".

TFIDF

What we are really trying to get at are keywords. A useful way of getting at keywords is to use TFIDF, that is Term Frequency (and) Inverse Document Frequency. The Term Frequency, which we already have, is the number of times a word appears within a document. Since some words occur quite a bit but are not particularly important in determining what a document is about (e.g., "said"), it is useful to prioritize unique words within the collection of documents as indicative of what each document contains. This is done through the IDF. However, since many rare words may not really be that important, one can combine TF andf IDF to score words according to how often they occur and their uniqueness. Thus, one can pick out words that truly are key for any particular document.

However, when we look at the top words by Chapter and TFIDF, we do not have any new discoveries. "God" is still the most important keyword. We also see "marvels" again, along with "said" and "say".

    WordUpper  Chapter       TFIDF
148       GOD      2.0  249.000000
48        GOD      4.0  212.000000
84        GOD      3.0  189.000000
44        GOD      9.0  156.000000
15    MARVELS     55.0  143.765171
92        GOD      5.0  137.000000
128      SAID     12.0  120.274019
971      SAID      2.0  117.023370
577      SAID      7.0  113.772721
784    JOSEPH     12.0  111.302068
448       SAY      6.0   99.788023
619    PEOPLE      7.0   93.973035
23        GOD     33.0   84.000000
368       GOD      6.0   83.000000
13      LORDS     55.0   81.999836

What if we normalize TFIDF? That is, we use the normalized word counts for TF in the TFIDF equation. Does this bring new insight? Yes. We see lots of new words in the top 15: "loser", "hates", "crusher", "plenty", "sacrifice", and "encourage". It is important to note that many of these come from the same chapters, 108 and 109. It would appear that these chapters have very strong keywords in them compared to the rest of the Quran's chapters.

    WordUpper  Chapter  NormalizedTFIDF
2       SERVE    109.0       736.307505
21    SHOCKER    101.0       627.502961
33      LOSER    108.0       582.681321
31      HATES    108.0       582.681321
15    MARVELS     55.0       570.294902
20  CONDEMNED    111.0       562.494348
70     BEGETS    112.0       509.846156
82    CRUSHER    104.0       502.002369
4     MANKIND    114.0       490.658890
37     PLENTY    108.0       471.084614
35  SACRIFICE    108.0       471.084614
1     WORSHIP    109.0       458.559404
5    SECURITY    106.0       451.093997
3        EVIL    113.0       436.671046
60  ENCOURAGE    103.0       425.495780

How often is a word the most important keyword in a chapter in the Quran? We can find the maximum normalized TFIDF word for each chapter, and then by grouping on each distinct word, we can get a count for each keyword. In the plot below, we see that "God" is the most important keyword in 32 of the 114 chapters.

There are 73 distinct words that the most important keyword in a chapter. Most only occur once, in fact, only the top 5 shown below are the most important keyword in a chapter more than once. The only word we haven't seen yet is "ease".

 

Bigrams

Let's move on from single words to bi-grams, that is, a pair of consecutive words. From these we can get a better sense of the context or meaning of words, since we know what other words surround them. In the below code, I find the bi-grams after having removed the stop words. The top 30 most commonly occurring bi-grams are then plotted.

How might we interpret the bigrams below? Looking at the plot, we see the most common bi-gram is "heavens earth". We have seen both words previously. This may refer to "[the] heavens [and the] earth", which could be a way of expressing God's power and dominion over both "the heavens and the earth". "Everything [in the] heavens" "belongs [to] God", who is the "Lord [of the] worlds".

We also see "God gracious", "gracious merciful", and "forgiving merciful", suggesting that God is gracious and merciful and forgiving. Another aspect of God is that "God knows", perhaps referring to God's omniscience or knowledge of all things.

Finally, we see "day [of] resurrection", referring to end times and one's eternal status. For those who "turn away" and do not "believe God" or worship gods "besides God" there is "painful punishment" awaiting. Thus, one should "fear God" and "know God" and listen to "God['s] messenger" and the "good news". Such news may be especially directed at the "children [of] Israel", that is, the Jewish people.

Trigrams

Now let's look at trigrams, that is, consecutive groupings of three non-stopwords. Using the same process as above, we can see similar groupings of words and infer a similar overall message.

For example, we see "God [is] gracious [and] merciful", "[in the] name [of] God [the] gracious", and "God [is] forgiving [and] merciful". "God [is all] hearing [and all] knowing", "God [is] mighty [and] wise", and God "knows best". We may infer that God "created [the] heavens [and] earth" and "everything [in the] heavens [and] earth " belongs to God. We are called to "believe [and to do] good deeds" or "righteous deeds". We must "fear God [and] obey", and as a "people [we should] worship God." We will face "God [on the] last day" to be assigned a place of "abiding" "forever". Those that the "Lord[']s marvels deny" should heed these warnings.

Something new and very interesting is "Jesus son [of] Mary". This refers to Jesus, who instead of being considered divine as Christian's do, is one of the prophets of Islam but is not considered divine.

Sentiment Analysis

A common textual analysis is to look at sentiment. So we can ask, what is the sentiment over the course of the Quran? We can look at this by actual chapter ordering, as well as chronologically. I adapt from an approach located here: https://mran.microsoft.com/posts/twitter.html. In essence, for each chapter, I do a count of the positive words and a count of the negative words, find the difference, and return that difference as the sentiment score. While such an approach has obvious problems (e.g., "not funny" is counted as positive since "not" is ignored and "funny" is a positive word on its own), it is a useful starting point.

The list of positive and negative words I used can be found here: http://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar.

Now we are ready for analysis. First, what does the sentiment look like by traditional chapter order? We can see that there are wide swings in sentiment, but these stabilize as time goes on. Why? After some exploration, we can see that, since the Quran is roughly ordered by chapter length, there are more opportunities for positive or negative words to occur, and this makes the longer chapters more likely to have higher or lower sentiment.

To adjust for this, we can normalize the sentiment by chapter length, that is, by calculating the average sentiment per verse in the chapter. When we do so, the sentiment looks a lot more balanced. There are some low points in the middle and some high points at the end, but overall, the sentiment appears to go back and forth.

We can do the same thing chronologically. In the absence of normalization, it would appear that until about the 40th chronological chapter, the sentiment is fairly stable and mostly positive. About chronological chapter 40, the sentiment swings back and forth wildly before settling out again. This stability ceases right after the change from Mecca to Medina (the red vertical line) in which the sentiment swings to positive and negative extremes again. The lowest sentiment in the whole Quran appears near the very end in chronological chapter 112 (traditional chapter 5).

When normalized, the first part of the Quran in chronological order has the strongest sentiment, and it is mostly positive. The middle stabilizes and is almost flat until the transition from Mecca to Medina, when the strong sentiment re-emerges. While it is mostly positive, the lowest normalized sentiments occur in chronological chapters 104 and 105 (traditional chapters 63 and 58).

Which words are in the chapters that have the strongest sentiment, both positive and negative? We can see which chapters have the highest and lowest sentiment and normalized sentiment in the tables below.

On the positive side, traditional chapter 28 is a longer chapter and has words like "given", "land", "guidance", "know", and "resurrection". Traditional chapter 103 is very short and has the words "good", "gracious", "merciful", "patience". Traditional chapter 110 is also short, and has words like "praise", "merciful", "celebrate", and "victory".

On the negative side, traditional chapter 5 is a longer chapter and has words like "fear", "disbelieve", "punishment", "hostile", and "hunger". Traditional chapter 58 has words like "punishment", "Satan", "oppose", "estrange", and "lie". Traditional chapter 63 has words like "hypocrites", "repel", "deluded", and "evil".

Obviously, to really understand what these chapters are about and why they have very positive or very negative sentiment, one must read them to understand the context of the positive/negative words that determine the sentiment.

 

Conclusion

So what is the Quran really about? While the above analysis can't provide any nuanced or really specific conclusions, we can say that the Quran (and Islam) is generally about:

• God, who is gracious, merciful, forgiving, powerful, knowledgeable, and creator of the heavens and the earth
• God's instructions on how to live one's life in proper obedience and worship to God
• The eternal reward or punishment due to humans depending on how they live their lives in obedience or disobedience to God

In this, we can say that the Quran (and Islam) is in very general agreement with the other Abrahamic faiths of Judaism and Christianity in these general teachings of the religion. That is, at a very general level, these three faiths appear to be in agreement on the general nature of God and the purpose and destiny of humanity. However, a more detailed comparison of these religions would likely reveal great and important differences. For example, on the question of "who is Jesus?", each gives a very different and important answer.

While such differences should not be minimized or ignored, nor should they detract from the commonalities and agreement that these faiths have with each other. Hopefully, a greater understanding of these similarities and differences can lead to common cause and coordination in confronting the challenges the world faces today. My hope is that the above analysis has provided at least a small and initial step forward in that greater understanding, not just for me, but for you as well.

Text Mining the Bible

Introduction

What is the Bible about?  There are many ways one could answer this question, and even more ways one could go about finding an answer to the question.  One could read the Bible or read books about the Bible.  One could look at the scriptural text, the theological formulations, the historical narrative, or adopt another lens by which to make sense of what the book as a whole is all about.

In what follows, I offer a data science approach to begin to answer this question through the use of keyword relevance analysis and data visualization.  Such an analysis can provide quick insights into the key words and themes that can then be understood visually.  While extremely simplistic and nowhere near as rich, complete, or meaningful as actually reading the Bible or reading books on the Bible, such an analysis can provide guidance as to what is important.  I detail the method and show the results of this analysis in what follows.

The Method

I wrote a Python script that scraped the Biblical text book by book, chapter by chapter from Biblegateway.com.  I selected the NRSV translation and used the Catholic selection and ordering of books in the Bible.  Once the book text was parsed and cleaned (ok, not perfectly, but good enough for demonstration purposes), the whole book was submitted to the Alchemy API for keyword relevance analysis.  The output of the API was a keyword (or key phrase) (e.g., "God") with the relevance or importance of the keyword within the context of the book (e.g., 0.95).  The higher the relevance, the more important the keyword was.  After compiling the results, I had a list of about 50 keywords by relevance for each book of the Bible.

I took the results and brought them into Tableau for interactive filtering and visualization.  Screenshots below of the results are taken from the resulting workbook.

Here are some links for those who may be interested regarding the nuts and bolts of the analysis:

• BibleGateway:
• source for text
• GitHub
• The code for web scraping and submitting to Alchemy API.
• You will need your own API key.  Modify the code as necessary to suit your needs.
• Documentation for Alchemy API
• Tableau workbook

Here are some high level results of the analysis with some simple conclusions.

Results: Most Import Keywords

So what is the Bible about?  From the above method, we can look at keywords and phrases by total counts and relevance.  We'll look at these in turn.

Total Counts

What are the most common keywords?  With 73 books in this Biblical list, a keyword could have a total count of 73 if it was a keyword in every book of the Bible.  According to the Alchemy API, the top 10 keywords by total count in the Bible are:

1. God (48)
2. Lord (46)
3. people (44)
4. house (26)
5. things (22)
6. Lord God (22)
7. Israel (22)
8. son (21)
9. land (20)
10. king (19)

Not surprisingly, the Bible is about "God" or the "Lord".  It is also about a "people", namely, the Israelites, and the story of their establishment in the "land" of Israel.  Much of the Old Testament talks about the various rising and falling of "king"s and the establishment of the "house" of the Lord (i.e., the temple) or being a member or descendant of a "house" (e.g., the house of David).   As lineage is important, "son" is used a lot to explain ancestry, and in the New Testament, this word takes on a special meaning as part of the phrase "son of God". 

Here are the top 3 keywords (God, Lord, people) over the range of the Bible.  While not categorized as relevant in every book of the Bible, "God"/"Lord" is very relevant throughout most of the Bible.  The word "people", though less relevant on average, is also found throughout the Bible.  This would suggest that the Bible is fundamentally about "God" and the relationship of "God" to a "people".

Noticeably absent from this list, is "Jesus", whose top entry is "Lord Jesus Christ" tied at #12 with a total count of 15.

Maximum Relevance

What are the most relevant keywords?  A keyword may only occur a few times but be extremely relevant in context, while another word may occur more often but may not be relevant at all.  So while total count does give an indication of relevance, it is not exhaustive.  According to this Quora post, the Alchemy API calculates relevance by using word position, context of other words, how many times it is used, and other statistics.  However, the specific details are not documented anywhere that I could find.

What are the most relevant keywords/phrases?  I looked at the maximum relevance for a keyword in the whole Bible.  In order of most relevant, the top 10 keywords/phrases are:

1. Lord Your God (0.998)
2. God (0.996)
3. beloved speaks (0.992)
4. son (0.990)
5. savior Jesus Christ (0.989)
6. shall (0.989)
7. Lord (0.988)
8. Judas (0.988)
9. Jesus Christ (0.984)
10. Holy Spirit (0.984)

Again, we can infer that the Bible is about "God", or more specifically (as addressed to the Israelites) the "Lord Your God".  "Beloved speaks" is interesting in that it comes from the Song of Solomon, in which Solomon writes poetry to his "beloved".  We also see again the importance of "son".  But now we also see "Jesus Christ" enter into the picture as important, as is the "Holy Spirit".  Thus, each person of the trinity is considered relevant.

The occurrence of "Judas" is also very interesting.  However, it is not a reference to the Judas who betrayed Jesus, but a reference to "Judas Maccabeus" from the book of 1 Maccabees.  The word "shall" comes from the book of Micah, who is a prophet and speaks of many things that "shall" happen.

Here are the top 5 (Lord Your God, God, beloved speaks, son, savior Jesus Christ).  We have seen that "God" is used throughout the Bible.  "Lord Your God" is used primarily in the early part of the Old Testament and then again in the prophets.  The word "son" is used heavily in the historical books of the Old Testament, and then again in the 4 gospels: Matthew, Mark, Luke, and John.  "Savior Jesus Christ" only makes one appearance in 2 Peter.

Total Counts and Maximum Relevance

We have seen that some words have high counts but lack relevance, while others have relevance but lack high counts.  Which keywords have both?  I increased the Total Counts filter and the Max Relevance filter until I had 10 keywords.  Choosing different thresholds would result in a slightly different combination, but my selection requires each keyword to have a max relevance above 0.90 and a Total Count greater than 12.    This results in the following list:

• Father
• God
• Holy Spirit
• Israel
• Jesus Christ
• King
• Lord
• Lord God
• Lord Your God
• Son

We have seen most of these already.  The exception is "Father".  It is a keyword 13 times and has a max relevance of 0.933.  It occurs throughout the Bible, but reaches a high point in Matthew.   Matthew opens up with a genealogy of Jesus in the form of "X was the father of Y, and Y the father of Z, and Z...", and this goes on for many lines. 

Most Relevant Keyword In Book Total Counts

Which words are consistently the most relevant in a book?  If we find the most relevant word in a book and then find the total counts of these words, which ones are on top?  The top 8 (those occurring more than once) are:

1. Lord (17)
2. God (14)
3. Son (4)
4. Christ (4)
5. King (4)
6. Christ Jesus (3)
7. Jesus Christ (3)
8. Jesus (2)

Again, "Lord" and "God" are consistently the most important.  We have also seen "son" and "king" before, as well as "Jesus".  Of note is that "Christ" now appears on its own in addition to being attached to Jesus as we have seen before.  The word "Christ" means "Messiah" who is the awaited savior and king of the Jews.  So there is the notion that Jesus is the "king" and savior of the Jewish people.

Keyword Category Total Counts

Many of these keywords could be further categorized or similarly grouped.  For example, "Jesus Christ" and "Christ Jesus" could be grouped with "Jesus".  After performing such groupings, what are the total counts now?  The top 10 are:

1. God (228)
2. Lord (150)
3. Family (149)
4. People/Israel (146)
5. Jesus (117)
6. Virtue (102)
7. Covenant/Law (61)
8. Body Part (58)
9. Land/Earth (56)
10. Humans (51)

"God" and "Lord" are still at the top, but next we have familial terms like "father", "son", "brother", "sister", and "child".  Then we have terms related to "Israel" or "People".  Next, any terms related to "Jesus".  After that, terms of virtue: peace, love, mercy, faith, hope.  Covenant/Law refers to terms related to the Old Testament law: priests, offering, covenant, and law.  Body Part refers to a mention of a body part like hands, heart, and eyes.  Land/Earth refers to the land, Earth, world, hill country, or some other generic description of location.  Lastly, humans refers to any non-familial term for humans: woman, man, and young men.

So we can see that this grouping still characterizes the Bible as about God, but perhaps it is about a relationship to people spoken of in largely familial concepts.  At the very least, family relationships are important and spoken of often.  Jesus is important too (more on that below).  A life of virtue in reference to (or perhaps in contrast with) the law/covenant of Israel is how one ought to live.  Such a vision has a broad scope, applying to the land and the Earth and humans of every kind on the Earth.

Specific Comparisons

Other specific comparisons could made of keywords and categories, but here are two interesting ones I saw.

Trinity: God/Lord, Jesus, Spirit/Holy Spirit

As we know, "God"/"Lord" is important throughout the Bible.  Once entering the New Testament, "Jesus" becomes equally important.  "Holy Spirit" has lesser importance, but is still prominent in the New Testament.  It is highest in the book of Acts, when it is more relevant than either "Jesus" or "God".

Virtue, Covenant/Law, Sin

The categories of Covenant/Law and Virtue seem to trade off in relevance.  In the beginning of the Bible, Covenant/Law keywords are most relevant.  In the prophetic books, Virtue keywords are more relevant.  In the beginning of the New Testament, both are important, but Covenant/Law is emphasized over Virtue. For the rest of the New Testament, however, Virtue is most important.  "Sin" appears here and there, but is only ever about as relevant as Virtue.

Other Insights?

While I could go on and on with more insights, I will leave that to you!  I have embedded the Tableau dashboard below.  You can also go here to view and use the dashboard.

Conclusion

So what is the Bible about?  From this simplistic keyword and relevance analysis, we can sum up the above findings and say that the Bible is broadly about:

• God (the Lord)
• The history of God's interaction with the people of Israel through the use of law and a covenant.
• The coming of Jesus, who is considered to be the Christ and king of the Jewish people, followed by the coming of the Holy Spirit.
• A shift towards a life focused on living by virtue instead of living (merely) by the law.

Much more can obviously be said here, but I think the main goal has been achieved.  That is, I have shown that a keyword and relevance analysis using data science methods (e.g., web scraping, APIs, data visualization) can reveal much about the important themes in a text, in particular, the Bible.

Buy or Sell? Stock Market Daily Tracker

With my life getting busier and busier, adding another item to the daily morning routine, no matter how short in the amount of time it takes, is to be avoided.  So as I have been observing the recent increases in the stock market and deciding that I should be paying attention more, I have been pondering about how I might do this in a more automated fashion.

Consequently, I decided to write a simple Python program that pulls some daily stock market information (e.g., today's opening, yearly high, yearly low), runs some calculations (e.g., opening / yearly high), and then after some simple conditional logic, tells me whether I should buy. This code runs on a daily schedule and opens up in a pop-up window automatically.

It is rather simplistic, but a quick glance at this every day is a simple way to keep an eye on the market and to be informed of action I should take in response.

The code is posted to GitHub if you are interested.

POS News Update: Best of 2016

In a previous post, I described an experiment with automating Twitter posts using news headlines with the nouns switched around.  I have been auto-Tweeting these posts since July 2015.  I did a previous "best of" post for 2015.  Now here are my 10 favorites from 2016 out of the close to 250 that were auto-Tweeted last year:

10. 'RARE NEWBORN REPORT MAY BE THE YOUNGEST EVER DETECTED'

• It is a newborn report on newborns.

9.  'POOL IS ENDING PARENT END BREEDING'

• Sorry parents, but no more breeding at your end of the pool.

8. 'CINCINNATI MAY PROLONG CHRONIC PAIN, CLINTON SUGGESTS'

• If the science is sound and you have chronic pain, you probably want to avoid Cincinnati in the future.

7. 'SPUDS GROW HUMAN SPIES OUTSIDE OF CHINA FOR 13 DAYS'

• For a long time humans have been growing potatoes.  Now potatoes are growing humans.  And not just any humans...

6. 'NORTH CHINA FIRES BALLISTIC PRESSURE INTO EATING'

• Speaking of China, its efforts to push eating are akin to warfare.

5. 'NEW YORK GHOSTBUSTER CAN ENFORCE PUMP FEARS ON MENUS, U SAYS'

• If you are a New York restaurant owner and your customers are feeling overly optimistic about the price of gasoline, who're you gonna call?

4. 'TWELVE MEAT RECOUNT PODS SHARE $43 MILLION IN [...] DENTAL JOB.'

• As many as twelve groups responsible for counting meat products profited from some very expensive dental work.

3. 'MOST NEW GIRLS SAY THEY'LL VOTE TO RE-ELECT THEIR MAGIC WILDFIRES'

• Despite what previous women have done, the younger generation is voting for mystical forest burning.

2. 'MICROSOFT'S 2016: DIPLOMATS 10 PRECAUTIONS AND INDIANS ARE THE BIG FEARS'

• Microsoft's biggest worries in 2016? What foreign representatives are saying, and Indians.

1.  'GET YOUR CANADA MANIPULATION WITH FITBIT 'SLEEP SCHEDULE''

• Finally, you can control Canada with a simple electronic device disguised as a sleeping aid.

Well, that was fun, but I am moving on to other kinds of coding experiments and automated Tweets.  So I'll be discontinuing the #POSNewsUpdate going forward.

Still, you can follow me on Twitter at @philanalytics to get the latest updates from the blog.

2016 NFL Game Predictions: Season Recap

The 2016 NFL season is over. How did I do in my predictions?  Here are the links to all predictions made:

2016 NFL Predictions:

• Superbowl: 1 - 0
• Conference Championships: 2 - 0
• Divisional Round: 1 - 3
• Wildcard Week: 2 - 2
• Week 17: 10 - 6
• Week 16: 10 - 6
• Week 15: 13 - 3
• Week 14: 11 -5
• Week 13: 8 - 7
• Week 12: 11 - 5 
• Week 11: 9 - 5
• Week 10: 9 - 5
• Week 9: 8 - 5
• Week 8: 6.5 - 6.5
• Week 7: 8.5 - 6.5
• Week 6: 8 - 7
• Week 5: 8 - 6
• Week 4: 7 - 8
• Week 3: 9 - 7
• Week 2: 11 - 5

Season Recap

Here are some summary statistics:

• Correct Prediction %:
• Whole season (regular and playoffs): 61%
• Regular season: 61%
• Playoffs: 55%
• Statistical significance two-tailed binomial test (95% CI):
• Whole season: 
• p-value:  0.0006
• CI: (0.55,0.67)
•  Regular season:
• p-value: 0.0006
• CI: (0.55,0.67)
• Playoffs:
• p-value: 1
• CI: (0.23,0.83)

My predictions were accurate about 61% of the time.  This was statistically significant.  I didn't do so well in the playoffs, but I am not surprised.  The playoffs are much harder to predict as the teams are better, more evenly matched, and there are fewer games to predict so each one counts for more in the percentages.

Comparison to FiveThirtyEight

In week 7 I compared my predictions to those being made by others.  Because 538.com took a similar approach to automating predictions and was doing the best at the time for all sites that I could find, I took 538.com to be a good benchmark to measure against. 

I should note that since then, the best overall predictions I have found have come from Elliot Harrison at NFL.com.  Assuming that he predicted the Patriots to win the Superbowl, he went 179-86-2 over the course of the whole season.  That is 68% correct predictions! 

This was not an automated approach so a direct comparison between what I have done/what 538.com has done and what Elliot Harrison has done is not entirely justifiable.  If we wanted to compare apples to apples, I would need to compare my own personal predictions (which may override my model's predictions) to what Elliot Harrison has done.  I did not keep track of this, but it is on my list to do for next season.

Consequently, I will focus on 538.  How did it do?

• Whole season: 64%
• Regular season: 64%
• Playoffs: 72%

538 did do better.  That's ok.  With more time and investment in this, I'd hope that I could match or exceed it's performance, but given other commitments, that hasn't been possible.  Maybe next season...

Season Visualized

Here is a chart that shows my progress over the course of the season.  You can see that I started well, but had some low points in weeks 4 - 8.  After that, apart from week 13, the model did pretty well and the season win ratio increased.  Playoffs were a bit erratic, mainly due to a poor division week prediction, but this didn't really effect the overall trend.

Here are comparisons of the weekly win ratios and season win ratios between 538 and me.  On a weekly basis, I beat 538 6 times, I was beat 8 times, and I tied 6 times.  So there was good back and forth on a weekly basis. 

However, 538 beat me by more and lost by less, so 538 had a higher season win ratio pretty much all the way through.

Going Forward

Just like last year, there is always next year.  I still have a long list of improvements to make, especially in code management and automation, and data management and pipeline.  We'll see what happens :)

Thanks for joining me this season.  See you next year!

2016 NFL Game Predictions: Best Team of the Season

The season is over and the Patriots have been crowned the SuperBowl champions.  But as I have asked in the past, which team was the "best" team this season?

Again, I use "best" in the sense that, according to my models, the team that is "best" would have the most wins in a tournament in which every team in the NFL played every other team in the NFL. There are 32 games total for each team (and playing yourself is a win), so the team with the most wins out of 32 would be the "best" team.  Note that my models assume that every player is healthy (i.e., no injuries).

In my previous analysis in week 14, the Cowboys, Seahawks, Chiefs, Broncos, and Patriots were the best (in that order) by expected value of wins when summing up probabilities.  Each of these teams was expected to win 21.x games out of 32.  By expected total number of wins (all or nothing),  the Cowboys, Chiefs, Broncos, Seahawks, and Raiders were the top 5 best.  The Patriots finished at number 6. 

What has changed since then?  Here are the final predictions of where each team would rank in a tournament of each team against every other team for the 2016 season:

 

Expected Value of Wins:

The Patriots are on top!  They moved up to 24.5 games won out of 32.  The Cowboys slide down to second at 23.9, while the Falcons take third at 22.8.  The Raiders, Chiefs, and Seahawks follow these.

The worst?  Browns finish last at 6.4 games out of 32.  The Browns are followed by the 49ers (7.2), Jaguars (7.4), Rams (7.6), and Bears (8.2) as being the worst teams this year.  Here are the full results:

Rank	Team	Expected Value
1	Patriots	24.544
2	Cowboys	23.926
3	Falcons	22.845
4	Raiders	22.799
5	Chiefs	22.729
6	Seahawks	22.281
7	Steelers	22.071
8	Giants	21.081
9	Lions	20.87
10	Dolphins	20.805
11	Packers	20.763
12	Texans	19.713
13	Titans	18.784
14	Redskins	18.438
15	Buccaneers	17.4
16	Colts	15.755
17	Ravens	15.566
18	Broncos	15.386
19	Cardinals	13.909
20	Saints	13.882
21	Vikings	13.881
22	Bills	13.386
23	Eagles	12.649
24	Bengals	11.664
25	Chargers	10.601
26	Jets	10.032
27	Panthers	9.274
28	Bears	8.242
29	Rams	7.628
30	Jaguars	7.437
31	49ers	7.237
32	Browns	6.422

 

Total Wins (All or Nothing):

With total wins, the Cowboys take the top slot as the only team predicted to beat the Patriots (52.2%) and to win all games.  Second are the Patriots, followed by the Raiders, Chiefs, Falcons, and Seahawks.  The worst are the Rams, followed by the Browns, Jaguars, 49ers, and Bears.  Here are the full results:

Rank	Team	Total Wins
1	Cowboys	32
2	Patriots	31
3	Raiders	30
4	Chiefs	29
5	Falcons	28
6	Seahawks	27
7	Giants	26
8	Steelers	25
9	Dolphins	24
10	Lions	23
11	Packers	22
12	Texans	21
13	Titans	20
14	Redskins	19
15	Buccaneers	18
16	Ravens	16
16	Colts	16
16	Broncos	16
19	Cardinals	14
20	Saints	13
21	Vikings	12
22	Bills	11
23	Eagles	10
24	Bengals	9
25	Chargers	8
26	Panthers	7
27	Jets	6
28	Bears	5
29	49ers	4
30	Jaguars	3
31	Browns	2
32	Rams	1

 

Conclusion:

I think it is safe to say that this year, the "best" team won the SuperBowl.  The only other contender for that title in my analysis was the Cowboys who finished second in expected value but first in total wins, trading spots with the Patriots.  It's too bad we didn't see a Patriots vs. Cowboys showdown in the SuperBowl.  But there is always next year...