Text Mining the Quran

Note: For the original jupyter notebook file, code, and data upon which this post is based, please visit my GitHub.

Introduction

I am interested in learning more about what Islam teaches and what exactly is in the text of the Quran(/Koran/Qu'ran), especially because of its current political relevance both in the United States and abroad. Consequently, to begin to do this, I will focus on text mining the text of the Quran. I intend to scrape the entire text from available sources online, grouping the text by chapter and verse. For each chapter, I will find the top words, keywords, and sentiment. I will also find meaningful clusters of words (i.e., bigrams and trigrams). These analyses will be visualized, along with any trends or patterns I find. Finally, I will summarize my conclusions.

I should note that my intentions in performing this analysis are purely descriptive with some inferences as to the meaning of my findings. I make no judgments on the truth of what the Quran contains or the correctness of its teachings, and nothing below should be construed as such. Also, as I am not a Quranic scholar, and as there are different traditions and methods of interpretation of the Quran, my conclusions below should be considered as an exploratory starting point in understanding the Quran, and not as representing any authoritative interpretation or conclusions.

The Quran

Before beginning, a quick introduction to the nature and structure of the Quran may be helpful. The Quran is the primary religious text used by Muslims, similar to the Bible (especially New Testament) for Christians and the Torah (the first five books of the Old Testament) for Jews. It contains revelations and instructions from God (in Arabic, "Allah") given to Muhammad, the prophet of Islam, from between 610 AD through 632 AD when Muhammed died (reference: https://en.wikipedia.org/wiki/Islam). The original and authoritative text is in Arabic, although there are translations into other languages as well (e.g., English). However, these would not be considered authoritative, and, as is a danger in all translations, may not convey the full and intended meaning due to cultural, textual, and historical differences. The Quran contains 114 Chapters ("Surahs") and is organized roughly according to length, from the longest chapter to the shortest chapter. However, the chronology of the chapters is also known, so one can also analyze chapters in their historical order, which I will do so below.

Packages

The below modules include the packages I will be using for this project. In particular, I will use:

• Beautiful Soup - text extraction
• re - regular expressions
• numpy - arrays
• pandas - data storage and transformation
• matplotlib - data visualization

Get Text

The below code blocks get the text from the chosen website (http://www.clearquran.com). The first three define functions that:

• get the website links
• get the website HTML
• get the text from the website HTML

Note: I chose the website above because it was one of the first returned in a search and because the structure of the website provided a relatively easy way to scrape and parse the data. I make no claim that this translation is better or more authentic than other English translations. Consequently, any conclusions I derive from the text depend on the accuracy of the textual translation itself.

The fourth code block uses the three functions above to get the website link for each chapter ("Surah") of the Quran, get the HTML for the website, get the text for each website (i.e., the verses for the chapter), clean the text (e.g., remove punctuation, numbers), split the words, do additional clean up, and then assign the chapter number, verse number, and words to a dataframe. The final result is a dataframe of Chapter, Verse, and Word for all of the words in the Quran.

The data is then saved to a CSV file for future use.

Clean Up

Before analysis, we need to clean up the data. I add column names to the dataframe and remove stopwords. Before removal, there are 146,279 words. After removal, there are 63,145 words.

Analysis

Word Counts

Now we can analyze the data. First, we can see that, indeed, the chapters are roughly organized from longest to shortest by looking at word counts for each chapter. There is a short introductory chapter, but this is followed by the longest chapter. Apart from occasional dips downward and then back up, the overall trend is from longer chapters to shorter chapters.

How does this compare with the Chronological Ordering of the chapters? Below, we first read in the chronological ordering from a website. Then, we compare the Chapter order with the Chronological order using the number of verses. As you can see, they are very different.

The chronological order begins dipping right at the beginning, but then climbs to an average maximum about 50 chapters in. This decreases on average until the upper 80s when there are several very large spikes, including the longest chapter of the Quran, before continuing a downward trend. There is a spike in long verses at the end as well. It is reasonable to conclude that these spikes relate to important periods in Muhammad's life when longer revelations would be necessary. For example, it would make sense to have longer revelations prior to Muhammad's death to guide the people after he had died.

The spike in the upper 80s appears to be related to the location of the revelation. The spike occurs at chronological order 87 (marked by a green vertical line), and corresponds to the first revelation in Medina following Muhammad's leaving Mecca. Such a lengthy revelation would be necessary to guide the community of believers in a new way of life in a new town, away from the persecution faced in Mecca.

Term Frequency

On to the actual text. Which words occur the most? When we look at term frequency over the whole of the Quran, we see that the word "God" ("Allah") occurs the most, over 2500 times. "Lord" is second with just under 1000. We also see "people", "believe", "earth", "merciful", "messenger", "punishment", "heavens", and "truth".

From this, we can infer that the Quran is foremost about God, whose most commonly ascribed attribute is mercy. It is addressed to a people or people in general, calling them to believe in God's truth, which is conveyed via a messenger (i.e., Muhammad). For those who believe, heaven awaits. For those who do not, punishment.

Normalized Term Frequency

Suppose we look at individual chapters. To compare across chapters, we should normalize the word counts within a chapter by dividing a word's total counts in a chapter by the chapter's total word count. The below plot shows the top 30 word-ratios, along with the associated word and chapter. We can see that "God" appears quite a bit. Other interesting words include "worship", "serve", "evil", "deny", "marvels", "condemned", and "shocker".

TFIDF

What we are really trying to get at are keywords. A useful way of getting at keywords is to use TFIDF, that is Term Frequency (and) Inverse Document Frequency. The Term Frequency, which we already have, is the number of times a word appears within a document. Since some words occur quite a bit but are not particularly important in determining what a document is about (e.g., "said"), it is useful to prioritize unique words within the collection of documents as indicative of what each document contains. This is done through the IDF. However, since many rare words may not really be that important, one can combine TF andf IDF to score words according to how often they occur and their uniqueness. Thus, one can pick out words that truly are key for any particular document.

However, when we look at the top words by Chapter and TFIDF, we do not have any new discoveries. "God" is still the most important keyword. We also see "marvels" again, along with "said" and "say".

    WordUpper  Chapter       TFIDF
148       GOD      2.0  249.000000
48        GOD      4.0  212.000000
84        GOD      3.0  189.000000
44        GOD      9.0  156.000000
15    MARVELS     55.0  143.765171
92        GOD      5.0  137.000000
128      SAID     12.0  120.274019
971      SAID      2.0  117.023370
577      SAID      7.0  113.772721
784    JOSEPH     12.0  111.302068
448       SAY      6.0   99.788023
619    PEOPLE      7.0   93.973035
23        GOD     33.0   84.000000
368       GOD      6.0   83.000000
13      LORDS     55.0   81.999836

What if we normalize TFIDF? That is, we use the normalized word counts for TF in the TFIDF equation. Does this bring new insight? Yes. We see lots of new words in the top 15: "loser", "hates", "crusher", "plenty", "sacrifice", and "encourage". It is important to note that many of these come from the same chapters, 108 and 109. It would appear that these chapters have very strong keywords in them compared to the rest of the Quran's chapters.

    WordUpper  Chapter  NormalizedTFIDF
2       SERVE    109.0       736.307505
21    SHOCKER    101.0       627.502961
33      LOSER    108.0       582.681321
31      HATES    108.0       582.681321
15    MARVELS     55.0       570.294902
20  CONDEMNED    111.0       562.494348
70     BEGETS    112.0       509.846156
82    CRUSHER    104.0       502.002369
4     MANKIND    114.0       490.658890
37     PLENTY    108.0       471.084614
35  SACRIFICE    108.0       471.084614
1     WORSHIP    109.0       458.559404
5    SECURITY    106.0       451.093997
3        EVIL    113.0       436.671046
60  ENCOURAGE    103.0       425.495780

How often is a word the most important keyword in a chapter in the Quran? We can find the maximum normalized TFIDF word for each chapter, and then by grouping on each distinct word, we can get a count for each keyword. In the plot below, we see that "God" is the most important keyword in 32 of the 114 chapters.

There are 73 distinct words that the most important keyword in a chapter. Most only occur once, in fact, only the top 5 shown below are the most important keyword in a chapter more than once. The only word we haven't seen yet is "ease".

 

Bigrams

Let's move on from single words to bi-grams, that is, a pair of consecutive words. From these we can get a better sense of the context or meaning of words, since we know what other words surround them. In the below code, I find the bi-grams after having removed the stop words. The top 30 most commonly occurring bi-grams are then plotted.

How might we interpret the bigrams below? Looking at the plot, we see the most common bi-gram is "heavens earth". We have seen both words previously. This may refer to "[the] heavens [and the] earth", which could be a way of expressing God's power and dominion over both "the heavens and the earth". "Everything [in the] heavens" "belongs [to] God", who is the "Lord [of the] worlds".

We also see "God gracious", "gracious merciful", and "forgiving merciful", suggesting that God is gracious and merciful and forgiving. Another aspect of God is that "God knows", perhaps referring to God's omniscience or knowledge of all things.

Finally, we see "day [of] resurrection", referring to end times and one's eternal status. For those who "turn away" and do not "believe God" or worship gods "besides God" there is "painful punishment" awaiting. Thus, one should "fear God" and "know God" and listen to "God['s] messenger" and the "good news". Such news may be especially directed at the "children [of] Israel", that is, the Jewish people.

Trigrams

Now let's look at trigrams, that is, consecutive groupings of three non-stopwords. Using the same process as above, we can see similar groupings of words and infer a similar overall message.

For example, we see "God [is] gracious [and] merciful", "[in the] name [of] God [the] gracious", and "God [is] forgiving [and] merciful". "God [is all] hearing [and all] knowing", "God [is] mighty [and] wise", and God "knows best". We may infer that God "created [the] heavens [and] earth" and "everything [in the] heavens [and] earth " belongs to God. We are called to "believe [and to do] good deeds" or "righteous deeds". We must "fear God [and] obey", and as a "people [we should] worship God." We will face "God [on the] last day" to be assigned a place of "abiding" "forever". Those that the "Lord[']s marvels deny" should heed these warnings.

Something new and very interesting is "Jesus son [of] Mary". This refers to Jesus, who instead of being considered divine as Christian's do, is one of the prophets of Islam but is not considered divine.

Sentiment Analysis

A common textual analysis is to look at sentiment. So we can ask, what is the sentiment over the course of the Quran? We can look at this by actual chapter ordering, as well as chronologically. I adapt from an approach located here: https://mran.microsoft.com/posts/twitter.html. In essence, for each chapter, I do a count of the positive words and a count of the negative words, find the difference, and return that difference as the sentiment score. While such an approach has obvious problems (e.g., "not funny" is counted as positive since "not" is ignored and "funny" is a positive word on its own), it is a useful starting point.

The list of positive and negative words I used can be found here: http://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar.

Now we are ready for analysis. First, what does the sentiment look like by traditional chapter order? We can see that there are wide swings in sentiment, but these stabilize as time goes on. Why? After some exploration, we can see that, since the Quran is roughly ordered by chapter length, there are more opportunities for positive or negative words to occur, and this makes the longer chapters more likely to have higher or lower sentiment.

To adjust for this, we can normalize the sentiment by chapter length, that is, by calculating the average sentiment per verse in the chapter. When we do so, the sentiment looks a lot more balanced. There are some low points in the middle and some high points at the end, but overall, the sentiment appears to go back and forth.

We can do the same thing chronologically. In the absence of normalization, it would appear that until about the 40th chronological chapter, the sentiment is fairly stable and mostly positive. About chronological chapter 40, the sentiment swings back and forth wildly before settling out again. This stability ceases right after the change from Mecca to Medina (the red vertical line) in which the sentiment swings to positive and negative extremes again. The lowest sentiment in the whole Quran appears near the very end in chronological chapter 112 (traditional chapter 5).

When normalized, the first part of the Quran in chronological order has the strongest sentiment, and it is mostly positive. The middle stabilizes and is almost flat until the transition from Mecca to Medina, when the strong sentiment re-emerges. While it is mostly positive, the lowest normalized sentiments occur in chronological chapters 104 and 105 (traditional chapters 63 and 58).

Which words are in the chapters that have the strongest sentiment, both positive and negative? We can see which chapters have the highest and lowest sentiment and normalized sentiment in the tables below.

On the positive side, traditional chapter 28 is a longer chapter and has words like "given", "land", "guidance", "know", and "resurrection". Traditional chapter 103 is very short and has the words "good", "gracious", "merciful", "patience". Traditional chapter 110 is also short, and has words like "praise", "merciful", "celebrate", and "victory".

On the negative side, traditional chapter 5 is a longer chapter and has words like "fear", "disbelieve", "punishment", "hostile", and "hunger". Traditional chapter 58 has words like "punishment", "Satan", "oppose", "estrange", and "lie". Traditional chapter 63 has words like "hypocrites", "repel", "deluded", and "evil".

Obviously, to really understand what these chapters are about and why they have very positive or very negative sentiment, one must read them to understand the context of the positive/negative words that determine the sentiment.

 

Conclusion

So what is the Quran really about? While the above analysis can't provide any nuanced or really specific conclusions, we can say that the Quran (and Islam) is generally about:

• God, who is gracious, merciful, forgiving, powerful, knowledgeable, and creator of the heavens and the earth
• God's instructions on how to live one's life in proper obedience and worship to God
• The eternal reward or punishment due to humans depending on how they live their lives in obedience or disobedience to God

In this, we can say that the Quran (and Islam) is in very general agreement with the other Abrahamic faiths of Judaism and Christianity in these general teachings of the religion. That is, at a very general level, these three faiths appear to be in agreement on the general nature of God and the purpose and destiny of humanity. However, a more detailed comparison of these religions would likely reveal great and important differences. For example, on the question of "who is Jesus?", each gives a very different and important answer.

While such differences should not be minimized or ignored, nor should they detract from the commonalities and agreement that these faiths have with each other. Hopefully, a greater understanding of these similarities and differences can lead to common cause and coordination in confronting the challenges the world faces today. My hope is that the above analysis has provided at least a small and initial step forward in that greater understanding, not just for me, but for you as well.

Text Mining the Bible

Introduction

What is the Bible about?  There are many ways one could answer this question, and even more ways one could go about finding an answer to the question.  One could read the Bible or read books about the Bible.  One could look at the scriptural text, the theological formulations, the historical narrative, or adopt another lens by which to make sense of what the book as a whole is all about.

In what follows, I offer a data science approach to begin to answer this question through the use of keyword relevance analysis and data visualization.  Such an analysis can provide quick insights into the key words and themes that can then be understood visually.  While extremely simplistic and nowhere near as rich, complete, or meaningful as actually reading the Bible or reading books on the Bible, such an analysis can provide guidance as to what is important.  I detail the method and show the results of this analysis in what follows.

The Method

I wrote a Python script that scraped the Biblical text book by book, chapter by chapter from Biblegateway.com.  I selected the NRSV translation and used the Catholic selection and ordering of books in the Bible.  Once the book text was parsed and cleaned (ok, not perfectly, but good enough for demonstration purposes), the whole book was submitted to the Alchemy API for keyword relevance analysis.  The output of the API was a keyword (or key phrase) (e.g., "God") with the relevance or importance of the keyword within the context of the book (e.g., 0.95).  The higher the relevance, the more important the keyword was.  After compiling the results, I had a list of about 50 keywords by relevance for each book of the Bible.

I took the results and brought them into Tableau for interactive filtering and visualization.  Screenshots below of the results are taken from the resulting workbook.

Here are some links for those who may be interested regarding the nuts and bolts of the analysis:

• BibleGateway:
• source for text
• GitHub
• The code for web scraping and submitting to Alchemy API.
• You will need your own API key.  Modify the code as necessary to suit your needs.
• Documentation for Alchemy API
• Tableau workbook

Here are some high level results of the analysis with some simple conclusions.

Results: Most Import Keywords

So what is the Bible about?  From the above method, we can look at keywords and phrases by total counts and relevance.  We'll look at these in turn.

Total Counts

What are the most common keywords?  With 73 books in this Biblical list, a keyword could have a total count of 73 if it was a keyword in every book of the Bible.  According to the Alchemy API, the top 10 keywords by total count in the Bible are:

1. God (48)
2. Lord (46)
3. people (44)
4. house (26)
5. things (22)
6. Lord God (22)
7. Israel (22)
8. son (21)
9. land (20)
10. king (19)

Not surprisingly, the Bible is about "God" or the "Lord".  It is also about a "people", namely, the Israelites, and the story of their establishment in the "land" of Israel.  Much of the Old Testament talks about the various rising and falling of "king"s and the establishment of the "house" of the Lord (i.e., the temple) or being a member or descendant of a "house" (e.g., the house of David).   As lineage is important, "son" is used a lot to explain ancestry, and in the New Testament, this word takes on a special meaning as part of the phrase "son of God". 

Here are the top 3 keywords (God, Lord, people) over the range of the Bible.  While not categorized as relevant in every book of the Bible, "God"/"Lord" is very relevant throughout most of the Bible.  The word "people", though less relevant on average, is also found throughout the Bible.  This would suggest that the Bible is fundamentally about "God" and the relationship of "God" to a "people".

Noticeably absent from this list, is "Jesus", whose top entry is "Lord Jesus Christ" tied at #12 with a total count of 15.

Maximum Relevance

What are the most relevant keywords?  A keyword may only occur a few times but be extremely relevant in context, while another word may occur more often but may not be relevant at all.  So while total count does give an indication of relevance, it is not exhaustive.  According to this Quora post, the Alchemy API calculates relevance by using word position, context of other words, how many times it is used, and other statistics.  However, the specific details are not documented anywhere that I could find.

What are the most relevant keywords/phrases?  I looked at the maximum relevance for a keyword in the whole Bible.  In order of most relevant, the top 10 keywords/phrases are:

1. Lord Your God (0.998)
2. God (0.996)
3. beloved speaks (0.992)
4. son (0.990)
5. savior Jesus Christ (0.989)
6. shall (0.989)
7. Lord (0.988)
8. Judas (0.988)
9. Jesus Christ (0.984)
10. Holy Spirit (0.984)

Again, we can infer that the Bible is about "God", or more specifically (as addressed to the Israelites) the "Lord Your God".  "Beloved speaks" is interesting in that it comes from the Song of Solomon, in which Solomon writes poetry to his "beloved".  We also see again the importance of "son".  But now we also see "Jesus Christ" enter into the picture as important, as is the "Holy Spirit".  Thus, each person of the trinity is considered relevant.

The occurrence of "Judas" is also very interesting.  However, it is not a reference to the Judas who betrayed Jesus, but a reference to "Judas Maccabeus" from the book of 1 Maccabees.  The word "shall" comes from the book of Micah, who is a prophet and speaks of many things that "shall" happen.

Here are the top 5 (Lord Your God, God, beloved speaks, son, savior Jesus Christ).  We have seen that "God" is used throughout the Bible.  "Lord Your God" is used primarily in the early part of the Old Testament and then again in the prophets.  The word "son" is used heavily in the historical books of the Old Testament, and then again in the 4 gospels: Matthew, Mark, Luke, and John.  "Savior Jesus Christ" only makes one appearance in 2 Peter.

Total Counts and Maximum Relevance

We have seen that some words have high counts but lack relevance, while others have relevance but lack high counts.  Which keywords have both?  I increased the Total Counts filter and the Max Relevance filter until I had 10 keywords.  Choosing different thresholds would result in a slightly different combination, but my selection requires each keyword to have a max relevance above 0.90 and a Total Count greater than 12.    This results in the following list:

• Father
• God
• Holy Spirit
• Israel
• Jesus Christ
• King
• Lord
• Lord God
• Lord Your God
• Son

We have seen most of these already.  The exception is "Father".  It is a keyword 13 times and has a max relevance of 0.933.  It occurs throughout the Bible, but reaches a high point in Matthew.   Matthew opens up with a genealogy of Jesus in the form of "X was the father of Y, and Y the father of Z, and Z...", and this goes on for many lines. 

Most Relevant Keyword In Book Total Counts

Which words are consistently the most relevant in a book?  If we find the most relevant word in a book and then find the total counts of these words, which ones are on top?  The top 8 (those occurring more than once) are:

1. Lord (17)
2. God (14)
3. Son (4)
4. Christ (4)
5. King (4)
6. Christ Jesus (3)
7. Jesus Christ (3)
8. Jesus (2)

Again, "Lord" and "God" are consistently the most important.  We have also seen "son" and "king" before, as well as "Jesus".  Of note is that "Christ" now appears on its own in addition to being attached to Jesus as we have seen before.  The word "Christ" means "Messiah" who is the awaited savior and king of the Jews.  So there is the notion that Jesus is the "king" and savior of the Jewish people.

Keyword Category Total Counts

Many of these keywords could be further categorized or similarly grouped.  For example, "Jesus Christ" and "Christ Jesus" could be grouped with "Jesus".  After performing such groupings, what are the total counts now?  The top 10 are:

1. God (228)
2. Lord (150)
3. Family (149)
4. People/Israel (146)
5. Jesus (117)
6. Virtue (102)
7. Covenant/Law (61)
8. Body Part (58)
9. Land/Earth (56)
10. Humans (51)

"God" and "Lord" are still at the top, but next we have familial terms like "father", "son", "brother", "sister", and "child".  Then we have terms related to "Israel" or "People".  Next, any terms related to "Jesus".  After that, terms of virtue: peace, love, mercy, faith, hope.  Covenant/Law refers to terms related to the Old Testament law: priests, offering, covenant, and law.  Body Part refers to a mention of a body part like hands, heart, and eyes.  Land/Earth refers to the land, Earth, world, hill country, or some other generic description of location.  Lastly, humans refers to any non-familial term for humans: woman, man, and young men.

So we can see that this grouping still characterizes the Bible as about God, but perhaps it is about a relationship to people spoken of in largely familial concepts.  At the very least, family relationships are important and spoken of often.  Jesus is important too (more on that below).  A life of virtue in reference to (or perhaps in contrast with) the law/covenant of Israel is how one ought to live.  Such a vision has a broad scope, applying to the land and the Earth and humans of every kind on the Earth.

Specific Comparisons

Other specific comparisons could made of keywords and categories, but here are two interesting ones I saw.

Trinity: God/Lord, Jesus, Spirit/Holy Spirit

As we know, "God"/"Lord" is important throughout the Bible.  Once entering the New Testament, "Jesus" becomes equally important.  "Holy Spirit" has lesser importance, but is still prominent in the New Testament.  It is highest in the book of Acts, when it is more relevant than either "Jesus" or "God".

Virtue, Covenant/Law, Sin

The categories of Covenant/Law and Virtue seem to trade off in relevance.  In the beginning of the Bible, Covenant/Law keywords are most relevant.  In the prophetic books, Virtue keywords are more relevant.  In the beginning of the New Testament, both are important, but Covenant/Law is emphasized over Virtue. For the rest of the New Testament, however, Virtue is most important.  "Sin" appears here and there, but is only ever about as relevant as Virtue.

Other Insights?

While I could go on and on with more insights, I will leave that to you!  I have embedded the Tableau dashboard below.  You can also go here to view and use the dashboard.

Conclusion

So what is the Bible about?  From this simplistic keyword and relevance analysis, we can sum up the above findings and say that the Bible is broadly about:

• God (the Lord)
• The history of God's interaction with the people of Israel through the use of law and a covenant.
• The coming of Jesus, who is considered to be the Christ and king of the Jewish people, followed by the coming of the Holy Spirit.
• A shift towards a life focused on living by virtue instead of living (merely) by the law.

Much more can obviously be said here, but I think the main goal has been achieved.  That is, I have shown that a keyword and relevance analysis using data science methods (e.g., web scraping, APIs, data visualization) can reveal much about the important themes in a text, in particular, the Bible.

Visualizing Natural Family Planning

Introduction

My wife and I use Natural Family Planning (NFP).  In fact, full disclosure, my wife teaches NFP.  For those that may not know, NFP is a way to achieve or avoid pregnancy naturally by using a woman's natural signs and observations throughout her cycle to determine the couple's fertility on any given day.  No, it's not the rhythm method.  It is actually quite effective in avoiding and achieving pregnancy, it encourages mutual support and communication about sex between the couple, and it does not require the use of hormones or devices to be effective.  Sounds pretty good, huh?  For more information, click here.

The specific brand of NFP we use is called the Creighton Model FertilityCare System.  As part of using this system, the couple charts the signs/observations for every day of the cycle on a paper chart.  When the cycle is complete, a new row on the chart is used.  This is repeated over and over for every cycle.  You get the following when you have completely filled out a chart:

At some point, I realized that my wife's many charts of her cycles formed a great dataset.  I know for every day of her cycle for the past 3 years what her fertility is and what the specific signs and observations associated with that day are.  I thought to myself, couldn't I visualize this data in a useful way to look at, say, a given day's likelihood of being fertile?  What other sorts of information could I gather from the charts?

Below are my observations and visualizations of what I uncovered about our use of NFP.  Yes, I am doing this with my wife's full knowledge, consent, editing, approval, and blessing.  Thanks for asking.  Yes, this is extremely personal, but I think it is worthwhile for our own and other's understanding of NFP and its uses.

If you are squeamish about topics related to sex, menstrual cycles, bodily fluids, and so forth, you probably should just stop here.  I'll try to be as clinical and impersonal as possible so that we can focus on the data and it's implications.

Understanding the Data

For every day, a user of the NFP system records several aspects of her fertility observations, which are recorded in the following way (for understanding more fully what this means, click here):

• Color stamp:
• Red (menstrual flow, spotting), Green (dry), Yellow (non-fertile discharge), White-baby (mucus), Green-baby (dry day within peak plus 3)
• Number:
• 0,2,4: various forms of dry observations
• 6,8,10 - indicates the length of the mucus
• Letters:
• VL (very light), L (light), M (medium), H (Heavy.  Used on red days.
• C (cloudy), K (clear), L (lubricative), P (pasty), B (brown), Y (Yellow), G (Gummy).  Used on mucus or discharge days.
• xX or AD:
• indicates number of observations.  E.g., x3 indicates 3 times observed that day.  AD means all day.
• I: whether intercourse took place that day.

So, if you saw this:

 

 

That would mean, "on 5/4 there was fertile peak type mucus that was peak type that was stretchy, cloudy, clear, and lubricative, and it was observed all day."

 

I converted this information into rows with the following columns:

• Date
• Day of cycle
• Stamp
• Number
• Flow type (VL, L, M, H)
• Type (0,4,6,8,10)
• Letters (B, C, K, L, G, Y, P)
• Duration
• Intercourse
• PeakDay
• Avoiding?
• Pregnant?
• Nursing?
• PostPartum?
• PMS?
• Headache?
• Fertile?

There were 1,114 row (days) in the dataset, consisting of 36 cycles.

 

What can we learn from this dataset?

 

Analysis and Visualization

Time for the fun.  I'll be using this opportunity to explore Microsoft's latest Power BI offerings for visualization.

 

A typical cycle lasts about 26 days (26.33 days with standard deviation of 1.66).  The longest cycle is 29 days (excluding the cycle in which a conception occurred), or 31 (if we count up to the day of confirmed pregnancy).

 

 

 

Stamps

What does a typical cycle look like in this dataset?  Look at the graph below.  The day of the cycle is along the x-axis and the percentage of stamp type is along the y-axis.

 

Note: I couldn't rearrange the stacking order of the categories in PowerBI, so I used Excel for this chart.

 

The first five days are always menstrual flow.  Then this tapers off on days 6, 7, and 8 and gives way to either dry days (green) or fertile days (white).  The fertile days peak in days 11, 12, and 13 and then give way to discharge days (yellow) or dry days.  There are occasional spotting (red) days throughout and occasional fertile days near the end of the cycle. 

 

The rising and falling of each kind of stamp is more easily seen in the below chart.

 

 

 

Menstrual Flow

The first five days are always menstrual flow.  Then it declines to day 10.  The kind of flow changes over the course of this time period.  Usually, the first day is medium or heavy.  The second day is almost always heavy.  The third is usually medium.  Then we progress to medium or light, then light or brown, then very light or brown for the rest menstrual flow.  See the below chart.

 

 

So a typical progression could be medium, heavy, medium, light, very light, brown.

 

Type

Consider the ugly and poorly designed chart below of type of mucus.  This reflects how long the mucus can stretch.

 

Note: an example of what not to do in data visualization.

 

There is a lot going on and it is not easy to see (partly deliberate).  I'll bring out the important points and show the individual graph as well.

• 0: dry days are rare except in two peaks.  The first is days 7-12, which is between the menstrual flow and the mucus buildup.  The second is after ovulation has occurred (the mucus), resulting in dry days for the rest of the cycle.

 

• 2: relatively dry days are similar to 0 in exhibiting two peaks along the same lines, but with more counts.

 

• 4: starts off at day 4 relatively high, then dips down a little during the mucus buildup.  There is a little peak after the mucus build up.  None of the peaks are very strong.

 

• 6: for cloudy/pasty discharge there are two peaks.  The first is in days 5-8 (just after the menstrual flow but before any mucus).  The second occurs in days 15-18 (immediately after the mucus) and then steadily declines (giving way to dryer days).

 

• 8: this mucus observation becomes more and more common, from nearly nothing, to peaking around days 14 and 15 (immediately after the peak in the cycle) and then suddenly dropping to very few for the rest of the cycle.

• 10: this kind of mucus builds from nothing to peaking in days 10-13 and then declines again towards the end of the cycle.

 

 

The change from 2 peaks (0,2,4) to 1 peak (6,8,10) can be seen in the follow graph:

 

The color changes from light (0) to dark (10) as the type of mucus increases in value.  This is easier to see when I group 0,2,4 as light and 6,8,10 as dark:

Peak By Day

When does the cycle reach its peak?  On average, about15 days (14.62 with 2.09 standard deviation).  Along with the average length of a cycle, this results in a post-peak phase of about 11 days (11.57 with 2.08 standard deviation).  Below is a stacked area graph representing the counts of occurrences of the peak day (P) and days following the peak day (1-15) by the day of the cycle. 

 

 

 Note: I couldn't figure out how to reorder the legend to be P-15.  It treats it as categorical instead of numerical/linear.  Ideally, the P would be on the bottom of the stack, then 1, then 2, etc. through 15.

 

Since the above graph is a little difficult to understand (although perhaps intriguing), below shows only the peak day distribution:

 

Note: perhaps this area chart isn't the best option for displaying counts over time. 

 

 

 

PMS

PMS only seems to occur following the Peak day.  It increases in likelihood as one nears the beginning of the next cycle.  This should not be surprising as PMS is "pre-menstrual" and for many women seems to signal the coming of the next cycle.

 

Note: This chart is from Excel.  Power BI can't rearrange the X-axis appropriately.

 

Intercourse

When does intercourse occur?  Consider the below chart:



 

 

 

We see a general pattern that is similar to a pattern we have seen before.  A first peak at around day 7, then dropping down in the middle about day 14, and then another peak about day 23, and then falling again.  What is going on here? 

 

The peaks in intercourse correspond to infertile days, while the drop in the middle corresponds to the fertile days of a typical cycle.  And this should be no surprise for users of the system.  A couple will, when avoiding pregnancy (which may be most of the time), use infertile days for intercourse while avoiding fertile days.  This is easier to see in the following two charts which filter on the couple's use of the method: (1) avoiding a pregnancy and (2) not avoiding a pregnancy.

 

The first chart below shows the couple avoiding a pregnancy, and hence, only using the infertile days for intercourse.  This is why we see two peaks, corresponding to the infertile days in the cycle.

 

 

 

 

In this second chart, the couple is not avoiding a pregnancy, and hence, intercourse occurs throughout the cycle.

 

 

 

For a fuller picture, consider the following table:

Avoiding Pregnancy?	Fertile?	Intercourse	No Intercourse	Sum Of Row	Intercourse/Sum Of Row
No	Yes	28	50	78	0.36
No	No	51	127	178	0.29
Yes	Yes	7	394	401	0.02
Yes	No	171	285	456	0.38
Sum Of Column		257	856	1113	0.23

 

 

The couple has intercourse 23% of the time.  But this 23% is not equally distributed amongst fertile vs. infertile days and avoiding vs. non-avoiding days.  Starting with avoiding days, when the couple is fertile, about 2% of days are used for intercourse.  Note that this 2% would be an improper use of the system.  When infertile, 38% of the days are used for intercourse.  When not avoiding, the fertile days are used 36% of the time for intercourse; the infertile days are used 29% of the time.

 

Given the foregoing, we can calculate that 35% of the time, the couple has intercourse when the fertility is in line with the avoiding vs. not avoiding intention (that is, when excluding the days in which the couple is fertile but is trying to avoid a pregnancy).

 

Various sources online citing how often a couple has sex on "average" per year give numbers like 104, 112, 118, 124, 127.  So 29% to 35% of all days in a year.  Thus, the above is in the "normal" range.

 

Letters

I broke the Letters column into a column of "Yes" or "No" values for each letter - B, C, K, L, G, Y, and P.  For example, if "CKL" was recorded, then C, K, and L columns would have a "Yes" for that row, while the other letter columns would have a "No".  Let's look at each of these:

 

B

Brown spotting occurs usually during or right after the menstrual flow.  However, it also is known to occur at ovulation.  This is likely what happened on day 14.

 

 

 

C

Cloudy mucus/discharge increases to a peak about day 17 and then declines afterwards.  This peak is right after the Peak day.  The C is associated with the yellow stamp, so it is not surprising that we see it following in much the same pattern.

 

 

 

 

K

K (clear) type mucus is associated with the buildup to and the occurrence of the peak of the cycle (e.g., ovulation).  So it is not surprising that Ks peak around days 11-13, right before and at a typical Peak day.  They fall back down again to almost nothing afterwards.

 

 

L

Lubricative days are also associated with fertility, and hence, appear most often in the buildup to and around the Peak day of the cycle.  These then decline afterwards with a few signs following after the peak.

 

 

G

Gummy type discharge doesn't really show any pattern.  While rare, it is distributed fairly evenly across the middle of the cycle, avoiding the very beginning and very ends of the cycle.

 P

Pasty discharge shows two peaks and is typically associated with the yellow stamps.  The first peak occurs right after the menstrual flow and then declines to almost nothing before the Peak.  Then it builds back up to day 18 as the mucus gives way.  It then declines again towards the end of the cycle.

Y

Yellow discharge doesn't show much of a pattern, except perhaps two groupings: right after the menstrual flow and right after the Peak day and following.  But this is also a pretty rare occurrence.

 

 Fertility

 

Finally, we can look at the fertile vs. non-fertile days.  Since red days and white days are treated as fertile, along with Peak 0,1,2 and 3 days, the fertile days follow the pattern of the red and white days put together along with the days just after the peak.

 

 

 

Summary

 In short, we can see that this woman's typical cycle lasts about 26 days, with ovulation (peak day as an estimate) occurring around day 15.  The menstrual flow lasts for 5 days before giving way to the mucus build up towards ovulation.  After the peak, discharge gives way to dry days as the cycle winds down to its end.  Then the cycle repeats.  Fertile signs occur in a single peak in the middle of the cycle, while infertile signs form two peaks on either side of the fertile peak.

 

Because the couple is using the system most of the time to avoid pregnancy, intercourse follows the pattern of infertile days: a smaller peak before ovulation and a larger peak after ovulation.  When the couple is not avoiding a pregnancy, intercourse occurs throughout the cycle.

 



Conclusion: Why It Matters

So what?  Why is this important?  Here are some data-driven/analytics reasons that one can see from the above data and charts above:

• Predictability: a couple/woman that charts the cycle can get a sense of what is normal.  They/she can predict when the woman will be fertile or infertile based on past cycles. 
• Planning: Predictability is especially useful when trying to achieve or avoid a pregnancy.  One can determine when to have intercourse so as to increase the chances of a pregnancy occurring (useful for couples struggling with infertility) or to avoid a pregnancy.  The woman can also determine when her next cycle is likely to begin, and can prepare for her menstrual flow.  Finally, other aspects associated with the cycle (e.g., PMS, headaches, cramping) can be recognized as cyclical/hormonal and prepared for/addressed appropriately.
• Disease prevention and health awareness: a woman that knows what her typical cycle looks like can recognize an abnormal cycle, which can indicate various kinds of diseases or complications.  The woman's cycle is a great indicator of health and can point to hormonal imbalances, Polycystic Ovarian Syndrome (PCOS), abnormal bleeding, infertility, and many other health issues.
• Infertility treatment: a couple trying to achieve a pregnancy can pinpoint the most fertile days.  Furthermore, a trained health professional can use the information contained in the chart to assess the couple for various kinds of infertility issues (e.g., limited mucus) and recommend treatments to increase fertility.

In addition to the above considerations, here are some other more philosophical considerations that are not based on the above data:

• Hormone free, all natural, green, organic sex: the couple requires no birth control devices, barriers, hormones, pills, or chemicals for an effective method of achieving and avoiding a pregnancy.
• Ethical: some couples may be concerned about the ethics of the potential abortifacient mechanisms of birth control methods or in using assisted reproductive technologies, such as in vitro fertilization.  These natural methods avoid those ethical concerns for both avoiding and achieving a pregnancy.
• Communication: the system encourages and to a certain extent requires the couple to communicate about their sexual health and intentions with regards to achieving or avoiding a pregnancy.  It is a shared system in which all information is available to both the man and the woman.  Each month, the couple revisits questions related to achieving or avoiding a pregnancy to help the couple remain on the same page in regards to their sex and family life.

In short, there are both good philosophical and analytical reasons for using NFP.  If you are interested in finding out more about this, click here. 

In any case, the data generated by users of the system provides a fruitful and interesting area for exploration and discovery, not only about oneself, but also about health, fertility, and sex in general.