Philosophy of Analytics, Lesson Four: Where

Introduction

In lesson one I discussed the importance of keeping in mind the purpose of analytics when you are starting and designing and running an analytics project (the why of analytics).   In lesson two, I discussed the importance of understanding the who behind analytics: who is doing it and who is it for.  In lesson three, I answered the when question: when should one use analytics?  In this lesson, I turn to the where question: where is analytics important?

Where?

Where is analytics needed?  Referring to the purpose of analytics, it is needed anywhere that one needs to justify or change beliefs by using data to generate true beliefs so that one can make decisions to achieve one's goals.  And where is that? Places/domains where beliefs (though perhaps true) are not justified and places where beliefs are false and need to be changed in accordance with the data.

A Problem

I have a Masters in philosophy.  What academic philosophers actually do is research and write on various topics related to ethics, politics, religion, metaphysics (what is the nature of existence and reality), epistemology (how do we know things and justify our beliefs), science, language, and the mind.  The way philosophers write papers or develop theories is that they formulate arguments using logical forms of reasoning on their chosen topic, usually to counter the arguments made by others that have a different view of a topic.

The nature of most of these topics is not subject to empirical verification (i.e., data), and so arguments and reasoning must do the work of justifying a point of view and convincing others to adopt that view.  While there is often some progress in countering views and persuading others, most often, even with full understanding of the opponent's arguments and reasons, both sides remain unpersuaded by their opponents view.  Peter van Inwagen and David Lewis, perhaps the two foremost metaphysicians in the 20th century have had many exchanges with this sort of result.  Van Inwagen laments that "I have done all I can to communicate [my views] to Lewis, and he has understood perfectly everything I have said, and he has not come to share my conclusions."
This is abstract reasoning and argumentation at its best, and often it isn't enough to determine the truth of the matter. 

At its worst, argumentation devolves into a host of fallacies: insulting/verbally attacking an opponent (ad hominem), misrepresenting the opponent's view (straw man), threats to the opponent, appealing to others/authority, appealing to pity, appealing to ignorance, or slippery slopes.  These methods of argumentation often appeal to and use emotion to ground the argument's effectiveness instead of looking to reasons and evidence.  Such rhetoric is all too common in our experience of statements made by politicians and other leaders, who seek power above the truth.

Fortunately, the matters with which businesses and organizations deal with are usually not the sorts of abstract things that are impossible to gather data about.  Even intangibles like "job performance" and "friendliness" can be defined and measured in useful ways (read this book for how to do this).  Indeed, philosophers have taken to collecting data about philosophical views as a way of bolstering support for their views and to gain insight into the nature of these philosophical debates.

Does data solve all of our problems then?  Unfortunately, no.  Ever heard the phrase, "lies, damned lies, and statistics?"  Data can be cited by both sides of an issue, each cherry picking specific aspects of the data to draw support for their view.  In these instances, what may be stated as the facts is true to a certain extent, but it may misrepresent the whole picture or be used to draw false conclusions.  Each side looks for evidence to support their foregone conclusion or theory.

I have worked in situations where the data behind key metrics was so cleaned, filtered, caveated, and adjusted that I was left wondering whether the metric actually corresponded to any meaningful sense of reality or was useful in any way.  Often, it seemed that the data was transformed so as to give the appearance of meeting specific targets; it was abused to support false or misleading conclusions instead of used to reflect reality.

A Solution

What are we to do then in seeking after the truth, if reasoning and evidence is not enough to guarantee an arrival at the truth?  Here are some suggestions, that while not perfect, can help combat bias and make sure that the decisions you make are based on evidence that is a good model of reality:

• Experiment: the scientific method can help.  We start with the evidence and then test the evidence against hypotheses.  The adoption of AB testing (hypothesis testing) in the business world is a great application of this approach.  With the data you have, formulate a hypothesis, test the hypothesis, and then evaluate whether it is true or not.  Such an approach is becoming more possible in business scenarios, enabling us to rely on data and evidence to drive and support our theories, instead of shoehorning data to fit our theories and arguments.  Finally, accept the results; don't explain them away, even if they contradict your preferred theory or intuition.
• Present alternatives: when presenting the results, present also the alternative explanations and conclusions that could be drawn from the analysis.  Explain why you have chosen a particular explanation in favor of others, but do not omit the others altogether.  You want to present the whole picture to your stakeholders.
• Conservative:  not in the political sense.  Be conservative in the conclusions you draw.  Imagine alternative explanations that are different from yours, and if they are at all reasonable, limit the scope of your explanation to what is supported by the data (and run more experiments to test these other hypotheses).  Don't extrapolate beyond your data.
• Clarity and Transparency: as best as you can, be clear about how you are handling the data (e.g., cleaning, filtering, transforming), and be transparent about your processes for gathering, transforming, analyzing, and visualizing data.  Make sure that your stakeholders know how you are doing your work (to the level of detail needed) so that they understand how to use the results appropriately.
• Accountability/Socialize: be accountable and socialize your data and results.  Use internal auditing. Have other team members look at your code or perform sanity checks on the results.  Work with those holding opposing views to jointly look at and understand the data before publishing results.  Make the raw data available so that others can check the data and verify the results.
• Patience/Kindness: where analytics is needed most is probably where it will meet the most resistance.  These will be areas that are relying on argument, intuition, and perhaps some fallacies to push decisions.  Analytics will be seen as unnecessary at best or as threatening at worst.  Individuals in these areas will be more interested in their own power, prestige, and status than in seeking the truth of the matter and making effective decisions.  Instead of demonizing such individuals, practice patience and kindness with them.  Build bridges.  Work with those that are receptive to analytics so that those that aren't can see the fruits of your labor (and will want to take part).  Data wins in the end; truth wins in the end.  If you have good analytics, the results of the decisions made in accordance with your analytics will vindicate you.  Be patient and kind.
• Move on: if however your patience and kindness hasn't led anywhere, and you have no way to move forward in your analytics work and in bringing about evidence-based decisions, it may be time to move on.  Find an organization/business that does care about data and making decisions based on data.  Don't waste too much time trying to convince those whose beliefs are impervious to sound reasoning and conclusive evidence.  You won't convince them, and you will continue to be a part of an organization/business likely headed towards failure or irrelevance.  Instead, move your skills somewhere else where they will be appreciated and utilized.  Data still wins in the end, but that victory may need to take place somewhere else.
• Humility:  you are not perfect.  You have been wrong and you will be wrong again.  You will make bad decisions and mistakes in how you gather, transform, analyze, and draw conclusions from your data.  When you are called out or discover your mistake, own up to it.  Work to resolve your mistake and learn from the experience.  Listen to others, especially those that disagree with you, to learn their thoughts on the work you are doing so you make sure you are covering your bases and their concerns.  Socrates is quoted as saying that he is wise because he does not claim to know anything.  Someone famous once said something like "I can only be certain of my own views when I know the arguments of my opponents better than they do."  Be humble in your work, and you will be more successful (along with your analytics) in the long run.

Remember that you are seeking the truth.  Analytics is about representing reality correctly so that one can make successful decisions based on true beliefs.

Conclusion

Use analytics everywhere, especially where it is most lacking and needed the most.  That is, use analytics where beliefs (though perhaps true) are not justified or where beliefs are false and need to be changed in accordance with the data.  When important decisions related to goals are being made, good analytics can help make sure that good decisions are made so as to achieve those goals.