A story of statistical inference
Off the Podium - Volume 1
Statistical inference is at once simple to define and yet challenging to understand. It is inextricably connected to the philosophical term induction or inductive inference, and therefore to all learning, particularly learning by experience. For this reason, it is a shame to keep it relegated to that dreaded domain of “statistics” or “research methods” in anyone’s education. But since this is where people usually learn about statistical inference, and since this is perhaps the only place in a curriculum that allows an explicit discussion about both the narrowly focused (research) consideration of statistical inference to the much broader, more universal, consideration of inductive inference and learning through experiences, that is where my career has conditioned me to raise such points.
In my clinical inquiry course sequence I teach that statistical inference, specifically, is our attempt to use probability theory to quantify the uncertainty of inductive inferences. Something that very few students have noticed - but is quite intentional in this course sequence - is that an introduction and review of cognitive bias and fallacies immediately proceeds content on the forms of inference, probability theory and statistical inference. The reason is - quite honestly - that the habits of mind associated with understanding statistical inference are the best protection we currently have against the flaws and biases of such errors in reasoning. This is commonly attributed to the work of Kahnemann and Tversky (popularized in Michael Lewis, The Undoing Project and even implicitly in his earlier work Moneyball). However, it can easily go back much further and it is almost foolish to cite particulars - but I will - such as Sir Arthur Conan Doyle with Sherlock Holmes and Paul Meehl in his “Clinical versus Statistical Prediction” in 1954. For the (as of today) most current treatise on this idea that statistical inference offers the currently known best approach to protection against bias and irrational reasoning see: Noise, by Kahnemann, Sibony and Sunstein, in particular Part V: Improving Judgements, Chapter 19, Debiasing and Decision Hygiene.
For now I shall return to the well worn trail by way of definition:
Statistical inference is the process of using data analysis to infer properties of an underlying distribution of probability. (From Upton, G., Cook, I. (2008) Oxford Dictionary of Statistics, OUP, via Wikipedia).
Inferential statistics apply statistical inference to infer properties of an underlying distribution of probability. The goal is to understand the properties of a population from a sample. But please note. This is done when a researcher collects a sample for study; AND when a clinician gains experience from practice when they observe (dynamically and interactively) the comings and goings of patients and learn from those experiences. The clinician with “experience” is only capable of applying what they learned about patients A to patient B if there is a reason to believe what occurred in patients A applies to a population that patient B is also part of. If patient B, for whatever reason, is not part of the population that patients A are a part of (or represent), then there is no warrant (no justification) for assuming what was learned from patients A can be applied to patient B. This is commonly referred to as “external validity” or “generalizability” during the process of assessing research.
Patient A’s heart is slightly left of midline in their mediastinum. Patient is part of the population - “human” - therefore we assume we will find Patient B’s heart slightly left of midline of their mediastinum. That’s an inference. It is not yet inductive. It becomes inductive when it is not just Patient A, but when it is a set of Patient’s that we call A. From the set of Patient’s we call A we infer that another patient (Patient B) has the same property.
The benefit that research brings is systematic observation and an explicit attempt to control for extraneous factors that influence the inductive inferences. Also, by collecting data explicitly and keeping track of the data (experiences observed) the researcher can utilize probability as opposed to the myriad of biases that influence what the clinician remembers as part of their gaining of experience.
Before going along too quickly, let’s reconnect to the definition and break down data and data analysis. All inductive inferences (including learning from experience) includes data. All data is processed with analysis. Some analysis is explicit and done with statistics, some is explicit and done with reflection, some is implicit and well, I don’t know what to say about implicit data analysis other than the fact that I believe it is more problematic than statistics or reflection.
Basically, data analysis assumes data. Data are information that are gathered through sensation and perception in moments of time that are experienced. Data are empirical (and yes, data are plural (datum are singular)). Each moment of your existence you are sensing and perceiving, you are gathering data during experiences, you are learning (or not learning) from these experiences. Sometimes the learning is unconscious, sometimes it is conscious. It may be easier for now if I simply connect unconscious learning with conditioning - if that doesn’t help, think Pavlov - if that doesn’t help recall a time memories of your childhood swept into your mind from the smell of your mom’s baking, or the sounds of a song. But this is not a post about unconscious learning so that’s all I’ll say here. Kahnemann proposes different thinking processes - fast and slow (popular exposition in the aptly titled book: “Thinking fast, thinking slow”). A crude summary is that slow thinking is conscious, fast thinking is unconscious (or at least greatly influenced by subconscious processing of data previously learned). Fast thinking, decision making in a moment, doesn’t have to be conditioning (in fact it often is not). It can also include decision making that started off as slow (reflective, analytical) thinking and has become routine (heuristic based). In Heuristics and Biases (a professional, not popular, originator of the “fast vs. slow thinking” written by Kahnemann and Tversky) there is a stronger association between fast thinking (using heuristics) and bias, than between slow thinking and bias.
As I have come to believe, with each situation you experience (and think about, reflect) there are three things (I’m still looking for a better word than things…) going on. One: you are confirming that it is actually occurring (which includes your understanding of past events and some preconceived notions you have regarding likelihood that what you’re experiencing can occur, how it may (or may not) cohere to your understanding of reality, your worldview). Two: you are considering how what you are experiencing came to pass (what led up to this situation, this moment, this experience), which is also heavily dependent on your understanding of the past. And three: you are considering how what you are experiencing changes, if at all, what you know about reality (the world, cosmos, “population”). Basically - what you believe.
All three of these things involve statistical inference. The first two are attempts to understand the data (the information emerging from the experiences). That understanding relies on prior understanding of reality (including the imputation of understanding from other people). The prior understanding of reality has - in many ways - emerged via statistical inference (perhaps not explicitly, but definitely implicitly - meaning you may not know the probabilities associated with your past statistical inferences but you qualitatively believe things, even in relative terms, as being more or less likely, more or less probable and the much less favorable terminology more or less certain (since once you’re only more or less certain then you are uncertain and probability or likelihood become better words).
The third thing is inductive (statistical) inference directly and as understood in the definition above - to infer properties of an underlying distribution of probability to understand the properties of a population from a sample. The samples are your experiences (the particulars, concrete, situations). The properties of a population are your vision of reality (the universal, abstract, concepts).
The point being made above is that statistical inference is not limited to particular statistical tools, maths or algorithms. It is not limited to research studies. Rather, it is a widely engaging and pervasive process of thought, or reflection. Research and the data it gathers, the statistical processes being utilized, the statistical inference it performs are merely a subset of the full set of inductive inferences being made every day. Research has the benefit of being very explicit, by design. Research has the benefit of being very systematic, by design.
Research is simply the systematic arrangement of experiences. Research generates observations that result in data upon which we can perform data analysis (statistics) and from which we can make explicit (precise?) statistical inferences.
Seeing the connection between all experience and research as a systematic arrangement of experiences opens up a world of experience expansion. Reading a research study provides insight into someone else’s experiences, including a reflection on what was and can be learned from those experiences. Just as reading a story, a poem, an allegory, a parable, a biography, all provide insight into someone else’s experiences that then open up a world of experience expansion.
I recognized this early in my career. I recognized that while I had little experience - I had at my fingertips (at that point in the stacks of the journals in the library, now on my computer, iPad and phone) the collective experience of many people that had systematically arranged many experiences and they described them to me, and reflected on them in front of me (research), and also in books both on research and on a wide ranging set of topics. All writers are engaging - by necessity - in slow reflective thinking and sharing their experiences and their analysis of those experiences in one form or another.
Learning to read research is no different than learning to read any form of literature. There’s a language, a form and expectations about what can be learned. The more broadly you read the more easily can you perceive the connections. In all forms of reading there is a constant connection between the particular and the universal, whether about the particular with inferences to the universal (narrative), or about the universal which provides meaning to the particular (poetry), or an attempt to do both all at once (allegory, parable).
Despite anyone’s proclamation that they don’t get or do statistics, you are constantly doing inductive inference whether you admit to it, realize it, or succumb to its nuances. Statistical inference is simply a way of being explicit about inductive inferences.
The word count is approaching my cap of 2000 words, which according to the Substack estimates means people can read this in approximately 10 minutes. Any more time spent with me in one sitting may be hazardous to your health. But I hope that in small doses, these newsletters offer insight and inspiration.
In summary, read. Read often and widely, read regularly and thoughtfully. Saying you read slowly or have a hard time reading is no more an excuse to not read than telling your patient that has a hard time with an exercise that they should therefore stop exercising all together.
In Off the Podium - Volume 2, I’ll specifically discuss some lessons learned in my course with rolling dice: alea idacta est (the die is cast).
Final Disclaimer: As with most things, the need to produce something with a beginning and an end has precluded me from expanding on any number of the points made in this post and from engaging with the myriad of thinkers that I have had the pleasure of reading and discussing these topics (past and present, dead and alive) that have shaped my thoughts.