Collection 7.3

Sampling Errors

Advanced lesson on fallacies arising from improper sampling and generalization. Students learn to identify when samples are too small, biased, or unrepresentative, how selection processes can distort conclusions, and when generalizations extend beyond what the evidence supports.

What to Notice

  • Recognize when conclusions are drawn from insufficient sample sizes or unrepresentative samples
  • Identify various forms of selection bias that can skew data interpretation
  • Distinguish between legitimate generalizations and overgeneralizations that ignore relevant variation
  • Detect when survivor bias or cherry-picked examples create false patterns

Concepts in This Collection

F070

Biased Sample

Drawing conclusions about a population from a sample that is not representative of that population, where the sampling method systematically over-represents or under-represents certain characteristics. The bias is in the selection process itself.

1 of 23
F071

Selection Bias

Drawing conclusions from data where the selection process for inclusion or observation is correlated with the outcome being studied, creating a distorted view of reality. This occurs when the mechanism determining what gets observed is related to the phenomenon being investigated.

2 of 23
F072

Survivorship Bias

Drawing conclusions based only on cases that survived some selection process while ignoring those that did not, leading to overly optimistic or skewed assessments. This is a specific type of selection bias where visibility depends on survival or success.

3 of 23
F073

Overgeneralization

Applying a general rule or principle too broadly, to cases where exceptions exist or where relevant differences make the generalization inappropriate. This involves extending a pattern or rule beyond its legitimate domain of application without accounting for important variations.

4 of 23
F074

Overwhelming Exception

Drawing a general conclusion from an exceptional case or using an exception to establish a rule, while ignoring that the case's atypical nature makes it unsuitable for generalization. This is the reverse of overgeneralization: treating exceptions as if they were the rule.

5 of 23
F265

Persuasive Definition

Defining a term in a way that contains an implicit value judgment or favorable/unfavorable characterization, then using that biased definition to support a conclusion. The fallacy occurs when the persuasive element is smuggled into the definition rather than argued for openly.

6 of 23
F266

Oversimplified Cause

Attributing a complex phenomenon to a single cause when multiple factors are actually responsible. This fallacy reduces multifaceted causation to a simple one-factor explanation, ignoring the interplay of contributing causes, contextual factors, and systemic complexity.

7 of 23
F267

Proving Too Much

Offering an argument that, if valid, would prove not only the intended conclusion but also other conclusions that are clearly false, absurd, or that contradict the arguer's own position. The argument's scope extends beyond the target claim to encompass implications the arguer doesn't intend or accept.

8 of 23
F268

Proof by Verbosity

Attempting to establish a conclusion by overwhelming the audience with an excessive volume of words, arguments, citations, or technical jargon rather than by providing sound logical reasoning. The sheer quantity of material creates an impression of rigor while obscuring weak reasoning.

9 of 23
F269

Proof by Intimidation

Presenting an argument in such complex, technical, or sophisticated terms that the audience feels unable or afraid to question it, despite the underlying reasoning being potentially flawed. The intimidation comes from intellectual rather than physical threat, exploiting fear of appearing ignorant or unsophisticated.

10 of 23
F270

Perfect Solution Fallacy

Rejecting a solution or course of action because it doesn't solve the entire problem perfectly or has minor flaws, when no perfect alternative exists. This creates a false dichotomy between a perfect solution (often unattainable) and doing nothing, ignoring that partial improvements have value.

11 of 23
F271

Privileging the Hypothesis

Devoting disproportionate attention, resources, or credence to a particular hypothesis without sufficient prior justification for singling it out from the vast space of alternative possibilities. This involves treating one explanation as particularly worthy of investigation or belief when the evidence equally supports countless other hypotheses.

12 of 23
F272

Presentism

Judging historical figures, events, or practices by contemporary moral, social, or intellectual standards rather than understanding them within their own historical context. This involves anachronistically applying modern values, knowledge, or norms to evaluate past actions.

13 of 23
F273

Proving a Negative

Demanding that someone prove a negative claim (typically that something doesn't exist or didn't happen) when such proof is extremely difficult or impossible to provide, often while shifting the burden of proof away from the person making the positive claim. This exploits the asymmetry between proving existence (requires one example) and proving non-existence (requires exhaustive search).

14 of 23
F274

Post Hoc Medicine

Concluding that a treatment or intervention caused recovery simply because improvement followed the treatment, without considering natural recovery, regression to the mean, placebo effects, or other confounding factors. This is post hoc reasoning specifically in medical or therapeutic contexts.

15 of 23
F275

Parable of the Broken Window

Failing to account for opportunity costs by focusing only on visible, direct effects of an action while ignoring the unseen alternatives that are foregone. Named after Frédéric Bastiat's parable where breaking a window seems to create economic benefit (the glazier's business) but ignores what the window owner would have done with that money otherwise.

16 of 23
F276

Package Deal Fallacy

Insisting that a collection of distinct ideas, policies, or propositions must be accepted or rejected as an indivisible unit, when the components are actually logically independent and could reasonably be accepted or rejected separately. This creates a false choice between accepting everything or rejecting everything.

17 of 23
F277

Pathetic Fallacy

Attributing human emotions, intentions, or consciousness to nature, inanimate objects, or non-human entities, then using these attributed characteristics as if they were real properties that can explain behavior or support arguments. This goes beyond metaphorical language to actual reasoning based on the false attribution.

18 of 23
F278

Omnipotence Paradox

Using apparent logical contradictions in the concept of omnipotence (such as 'Can an omnipotent being create a stone too heavy for itself to lift?') to argue against the existence or coherence of omnipotent beings, without recognizing that the paradox may stem from incoherent formulations, category errors, or misunderstandings of omnipotence rather than genuine logical impossibility.

19 of 23
F279

Olfactory Rhetoric

Dismissing arguments, people, or ideas by associating them with bad smells, uncleanliness, or disease, exploiting the instinctive human disgust response to bypass rational evaluation. This involves using olfactory metaphors or literal claims about smell to marginalize rather than engage with substance.

20 of 23
F280

Overexplanation

Providing such extensive, detailed, or complex explanations for phenomena that the explanation becomes less credible, less understandable, or reveals motivated reasoning to preserve a preferred theory. This occurs when the desire to explain everything leads to ad hoc additions, unfalsifiable claims, or explanations more complex than the phenomena they explain.

21 of 23
F281

Paralysis of Analysis

Failing to make necessary decisions or take timely action because of excessive analysis, perpetual gathering of more information, or insistence on complete certainty before acting. This involves treating the lack of perfect information as justification for indefinite delay when reasonable decision-making is possible with available information.

22 of 23
F282

Passive Voice Fallacy

Using passive voice constructions to conceal agency, avoid responsibility, or obscure who performed an action, particularly when the actor's identity is relevant to evaluating the claim or assigning accountability. This involves strategic grammatical choices that make actions appear to happen without actors.

23 of 23