Unconventional Logic

Making you think outside of the crate

Information Validity

Lt. Col. Custer

          On the morning of June 25th, 1876, Lieutenant Colonel George Armstrong Custer was a confident, successful and well-respected commander in the United States Military as he marched into the valley between the Bighorn River and the small western fork called the “Little Bighorn”. With twelve companies of the 7th cavalry at his command and a brilliant battle plan in place, Custer was understandably confident of his impending route of the increasingly troublesome Natives. Lt. Col. Custer planned to evict a large settlement of Lakota, Cheyenne and Arapaho natives by splitting his force into three battalions and dispatching them from three forts to enveloping the settlement. There was only one problem with Custer’s plan, it was based on incorrect information. The events that followed would be discussed, dissected and dissertated throughout the years and ultimately go down as one of the greatest blunders of all time. The massacre of the five companies under his immediate command changed Custer from a tactical mastermind to the poster boy for bad decisions overnight. Custer’s chief folly was that he did not verify the information he used to draw his conclusions. His death, and the deaths of his troops, were not due to his stupidity or incompetence as a military commander, but by decisions made from bad information. In this article I will explain how three fundamental issues led to Custer’s downfall, how to identify these issues in information today, and how The Battle of Little Bighorn warns us of the dangers we face in the world today.

Unconventional Logic is all about logical conclusions derived from evidence. We are not here to tell you The Twelve Secret Life Hacks to Accomplish All of Your Dreams! or The Low-Down on Current Events and How You Should Feel About Them. We are here to provide opinions on important topics and support our opinions with evidence. We are here to provide our readers with that same evidence and to allow you to form your own opinions, even if they differ from our own. We want our readers to leave our site more capable and informed than when they entered, and with the ability to challenge the claims of other media sources. In order to achieve our mission, we feel that it is important to first talk about what evidence actually is. How do I know what is good information? How can I tell if what I hear or read is trustworthy? Why is any of this even important? To answer these questions we need to take a journey down the rabbit hole of logic; into the worlds of statistics, psychology, and academia. Let’s start at the beginning.

When we look at information and are trying to determine whether or not it’s good, we need to focus on three main areas where problems can arise: Data, Inference, and Application. If any one of these categories is flawed, then it has a marked effect on the validity of that information. Our late friend Lt. Col. Custard had issues with all three of these categories, and as a result more than half of the troops under his command lost their lives. Our individual choices and actions based on bad sources of information might not carry such drastic consequences, but enough people making decisions using the same bad info might have catastrophic results. That’s why it is so important that we as a society learn how to vet information by examining the validity of information, or it’s trustworthiness. For this purpose, I have developed a simple three step process to determining whether or not information is trustworthy. These steps are three questions that will help eliminate shoddy sources of information quickly and consistently, because if the answer to any one of these questions is no, then the information cannot be considered trustworthy.

Step 1: Ask – Are there sources backing this information?

Step 2: Ask – Are the sources reliable?

Step 3: Ask – Does this information use logic when drawing meaning from its sources?

These three steps will tell you with a relatively high degree of certainty if information can be trusted. Note that I didn’t say that this would tell you if the information is actually true or false, but rather that this approach to vetting info will tell you whether or not it should be trusted. Let’s look at an example to help clear this up.

Suppose you see an article which claims that the current leadership is failing and needs to be removed from power because there are 110 million unemployed people in America right now. This number shocks and scares you, so you spend the next 2 hours reading similar articles that all claim over 100 million unemployed citizens in the U.S.

Now let’s imagine that immediately after reading this article you utilize the three-step approach to vet the information provided by the article. You ask yourself, “Are there any sources?”. You see that the article cites the Bureau of Labor Statistics data from 2016. Step one complete, so far this information is still trustworthy.

Next you ask, “Is this source reliable?”. Well obviously the BLS is a credible source of data on employment statistics. They specialize in this type of research, act mostly autonomously, and provide all of their data points and data collection methods. Step 2 complete and this information still checks out.

Lastly, you ask “Is this a logical conclusion to draw from the data?”. That’s where the article hits a snag. Upon looking at the data, you see that the BLS reported approximately 157 million Americans that are employed, 94 million that cannot work or chose not to (such as children and retirees), and 16 million able-bodied Americans who want a job but do not currently hold one. A little deduction tells you that the article clearly bent the data to fit their conclusions by combining the actual unemployment numbers with the population that is not actually part of the workforce. So the answer to the third question is no, the conclusions drawn from the data are not logical. Therefore, this information is not trustworthy. Technically the information is true, but its application is false.

Information like in our example scenario illustrates the danger of the misinformation that is available to the public. This type of information is what leads to flat earth theories, climate change denial, and the idea that vaccines cause autism. We cannot make the progress we so desperately need when our society is misled so thoroughly, defenseless against the ocean of stupidity we call the internet. For us to grow, learn, and make rational decisions to shape our future we absolutely MUST be able to tell the Truth from the Lies.

My process is a great tool to protect yourself against untrustworthy information, but you need to be able to effectively answer the questions.

Let’s take a deeper look at each question to help us answer confidently and correctly.

Question #1: Are there any sources backing up this information?

This one is pretty easy to answer. When you watch the news, read an article, listen to a podcast, or see a tweet from a social media personality look for key phrases. “According to a study by X…”, or “The University of Y released a study that…”, or “Recent data from the Department of Z suggests that….”. These are indicators that the information you are being presented has a source that supports the info. Basically; No Source = Not Trustworthy.

Question #2: Is this source reliable?

To answer this, we need to look at both the data and the nature of the source. Start with the data. There are hundreds of issues that can arise with data collection. In the world of statistics, these issues would be referred to as threats to internal and external validity and reliability. Most data come from controlled experiments (Yes, the word “data” is actually plural, and I hate that grammatical rule too). In experimental design, researchers control independent variables and measure the effect this has on a dependent variable. Unfortunately, despite the best efforts and intentions of researchers, it is impossible to completely control a research environment. Outside variables, often referred to as confounding variables, inevitably get into the mix and can have unknown effects on the dependent variable. Now this may all sound like incredibly complex stuff, and that’s because it is. In future installations of this series I will delve more into the details of validity threats, confounders and experimental design.

For now, focus on these areas: Who, How, and Where from.

Who gathered the data? This is the “According to ____” part. Generally speaking, University studies, research centers, and government agencies are good gatherers of data. How did they gather it? This refers to the data collection method. Controlled studies with direct observation are best, surveys/polls/questionnaires can also be acceptable. Where did they get the data from? We need to know if the population that the data came from is representative of the population the data is being applied to. For instance, if a poll claims that a out of 78% of Americans believe that the most important right in the Bill of Rights is The Right to Bare Arms, but the survey population is comprised of 100 people living in Wyoming (the state with the most registered guns per capita according to the ATF’s National Firearms Registration and Transfer Record), then there is obvious sampling bias and the source cannot be trusted.

Question #3: Is this a logical conclusion to draw from the data?

The key to answering this question is in understanding the difference between Data and Evidence. Data are just observations that are representative of the sample population. Evidence is data that are both representative and supportive of a theory. Valid conclusions are theories that are supported by evidence. In our mock survey of Second Amendment supporters in Wyoming, the conclusion was based on data, not evidence. The data collected from the survey were representative of the population being sampled (100 Residents of Wyoming). By this I mean that the answers were definitely from the people who took the survey and were likely honest answers., However, the data did not support the theory that 78% of Americans prioritize the Right to Bare Arms over other rights because it was not representative of the population to which the theory was being applied (All Americans). This last step is easily the most complicated one to answer. Luckily, most information will not pass the first step, less will meet the criterion for the second step, which leaves only a very small amount of information that must be run through the third step.

Now that we have a method for verifying information and an understanding of how to apply this process, let’s go back to our old friend Lt. Col. Custer and see if we can find where he went wrong. Custer based his entire plan on three pieces of information: Reports on enemy numbers in the valley from his Native scouts, the known number of Natives that left the reservations to join forces with Sitting Bull, and his own experiences with Natives.

The first piece of information was his scouting reports. A handful of Native Scouts reported that they observed thousands of horses and anywhere from 800 to 1,500 warriors in the village. They gathered this information by sight from miles away atop a hill. Let’s run this through our process. Are there sources backing the claim of 800-1,500 warriors? Yes, several scouts reported direct observations of the number of enemy warriors. Are these sources reliable? (Who, How and Where From?) The Native Scouts are trusted by Custer and are more qualified to determine the size of Native forces from the data points they collected (size of the herd of horses). The method of collection was by eyesight from several miles away, making the reliability of collection highly questionable. The scouts took these data points not from seeing all of the actual warriors, but from indirect indicators such as the size of the herd and village. From these questions, this information does not pass the second step of our process and therefore cannot be considered as trustworthy. History would tell us we are correct.

The second piece of information was the known number of Natives that left reservations to follow sitting bull. Previous reports told Custer that roughly 800 Native Warriors left the reservations to join Sitting Bull, so he determined the size of the force he would face to be within a margin of error from that report. Again, let’s apply our process. Are there sources backing this data? Yes, reports from several Natives all reported approximately 800 warriors left the reservations. Are these sources reliable? (Who, How and Where From?) Native civilians, scouts and U.S. Army scouts all independently gave similar reports. The data points were collected by direct observation, often by Natives optimally positioned to collect the data (they were in the village that the warriors left from). The Data was collected by observing the warriors themselves, not indirect indicators such as horse tracks or old camp sites. Yes, the sources can be considered reliable. Are the conclusions logical? To answer this, you have to play Devil’s Advocate. Ask yourself, is it possible that this conclusion is wrong and how could that happen. It absolutely could be incorrect because there could be other Native warriors that joined Sitting Bull’s forces before or after the 800 warriors were observed leaving, or the warriors may not have joined Sitting Bull at all. So, there is a possibility that Sitting Bull’s forces are either larger or smaller than the conclusion would indicate. Therefore, the conclusion is not logical, because it does not eliminate rival theories, and as such cannot be considered trustworthy. As it turns out, several thousand other Natives, from many different tribes, joined Sitting Bull only a short time before the battle.

The last piece of information Custer used to make his conclusions was from his own experience with Native Americans. Particularly, Custer was relying on his observations of how Native Warriors would react to threats to their civilians, observations he made while engaging the Cheyenne tribes in Kansas nearly a decade earlier. Custer’s observations were that Native Warriors would likely try for peace if they felt there was a danger of their civilians being hurt, and that they would ultimately scatter into smaller groups if they felt that the threat was imminent and unavoidable. Once more, let’s put this information through our three-step process. Are there sources supporting this information? Yes, data collected from Custer’s earlier campaigns. Is this source reliable? It’s hard to doubt oneself, so no-one can fault Custer for trusting his own observations to be accurate (and in actuality they were not faulty). Is the conclusion logical? Here we run into the same issue from our mock survey, the data collected from the sample population (Cheyenne tribes in Kansas) were not representative of the population Custer was applying this information to (Lakota tribes in Montana). We also stumble across a new logical issue; Custer’s data was out-of-date. This confounder of data is called Maturation, where data collected from a population early on in a study may not be representative of the population at the time of application. Native American war tactics in general may have changed in the decade between Custer’s encounters with the Cheyenne. All-in-all, Custer’s information based on his experiences was not logical and therefore not trustworthy.

By applying our process of vetting information, we can see that all three of the information sets that Custer used to plan his campaign were not trustworthy. While the issues with the information were largely not Custer’s personal fault, his application suffered from not establishing validity.

Humans are becoming overpopulated, water is becoming scarcer, our ability to cultivate food becoming less able to satisfy demands, green space is shrinking while garbage piles grow. We need serious progress in science and technology, medicine, and human relations to overcome the staggering odds set against sustaining our current trajectory. We cannot do this without substantial support from the general population. Change takes money and new policy, both of which require people who understand the need for change. Right now, we are submersed in a torrent of false-truths stemming from political and idealistic squabbling, unable to focus on the real problems at hand. We spend far too much time, effort, and money on the lies because we cannot discern them from the truth.

So the next time you watch the news, remember the three-step process. Ask yourself if the information you are receiving is trustworthy. Remember, stupidity and lies are contagious, and the only remedy is logic and truth.

Resources

Connell, E. S. (1997). Son of the Morning Star: Custer and the Little Bighorn. Macmillan.

Downing, S. M., & Haladyna, T. M. (2004). Validity threats: overcoming interference with proposed interpretations of assessment data. Medical Education38(3), 327-333.

Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The qualitative report8(4), 597-606.

Greenland, S., & Robins, J. M. (1986). Identifiability, exchangeability, and epidemiological confounding. International journal of epidemiology15(3), 413-419.

Ihantola, E. M., & Kihn, L. A. (2011). Threats to validity and reliability in mixed methods accounting research. Qualitative Research in Accounting & Management8(1), 39-58.

Kish, L. (1959). Some statistical problems in research design. American Sociological Review, 328-338.

Shpitser, I., & Pearl, J. (2008). Complete identification methods for the causal hierarchy. Journal of Machine Learning Research9(Sep), 1941-1979.


Tagged , , , , , ,