The Texas Sharpshooter Fallacy in Bioinformatics: When Your Analysis Defines Its Own Discovery

The fallacy, in one image

A Texan empties a magazine into the side of a barn. He walks up, finds the spot where the holes cluster tightest, paints a bullseye around that patch, and calls himself a sharpshooter. See the trick? The target got drawn after the shots, around a pattern that noise was always going to produce somewhere. He fits the hypothesis to the data, then presents the whole thing as if he had been aiming true from the start.

Now look at your last dataset. Twenty thousand genes, thousands of cells or samples, sometimes millions of variants, every measurement soaked in technical and biological noise. That barn wall is in front of you on every project. And every time you let an algorithm find the tightest patch of holes, then draw your biological bullseye around it, the Texan is close by.

The whole thing comes down to one sentence: the trouble starts when the evidence you use to justify a result is the same evidence that produced it.

The core mechanism: double dipping

Here is the pipeline everyone runs, more or less, in single-cell:

1. Normalize, select highly variable genes, reduce dimensions, build a neighbor graph.
2. Cluster (Louvain, Leiden), and you get twelve clusters.
3. For each cluster, run a differential test against the rest to find "marker genes."
4. You marvel at p-values like 1e-87, annotate your clusters as cell types, and write it up.

Where is the Texan hiding? At step 3. The clusters from step 2 were defined to maximize exactly the expression differences you then "discover." Testing for them on the same data guarantees you spectacular significance, even when your cells come from one perfectly homogeneous population with no real structure at all.

This is a proven property of the procedure. Gao, Bien and Witten established that classical difference-in-means tests control the Type I error rate only when the groups are fixed in advance.¹ When the groups come out of clustering, that error rate blows up. The nastiest part: the inflation persists even if you use two independent datasets, one to define the clusters and the other to test them. Your usual instinct, "I'll just split my data," does not save you here.

The same authors point out that this trap lives inside tools you probably reach for every week: FindMarkers-style workflows compute marker p-values on the very data used to form the clusters.¹ Those values do their job just fine for ranking and describing clusters. Treating them as valid p-values for the hypothesis "this cluster is a real, distinct population" is the fallacy, dressed up as a respectable function call.

The conceptual ancestor of all this is Kriegeskorte and colleagues' now-famous warning about double dipping in neuroimaging: reusing the same dataset for selection and for selective analysis produces distorted descriptive statistics and invalid inference whenever the test statistic depends on the selection criterion under the null.² Swap "voxels" for "genes and cells" and the argument carries over unchanged.

Over-clustering, or the shrinking bullseye

The Texan has a second move. If his first bullseye impresses no one, he draws a smaller one. For you, that is the resolution knob. You bump the resolution, a cluster splits, and here comes a "novel rare subtype." A few markers later, a discovery is born.

Grabski, Street and Irizarry looked hard at this.³ The clustering algorithms in standard pipelines are heuristic and do not account for statistical uncertainty, and the common practice of picking the number of clusters via stability metrics can lead to overconfidence in the discovery of novel cell types. Applied to already-published analyses, their significance-testing framework flags concrete cases of over-clustering.

The lesson: with enough resolution, you can always carve a cloud of cells into groups that look tight. Does tightness prove anything, when tightness is exactly what you optimized for?

GWAS and the winner's curse

Change fields, the barn wall stays the same. An association study scans millions of SNPs. The ones that clear the threshold have, on average, an estimated effect stronger than their true effect, because you kept them precisely for posting the most extreme value. In replication, the effect shrinks. Your discovery set both chose the variants and measured their effect, which is the target drawn around the holes. The phenomenon has a name, the winner's curse, and dedicated statistical corrections exist (review in PLOS Genetics, doi:10.1371/journal.pgen.1010546).

Differential expression, then enrichment

Another rigged chain. You select your differential genes, feed the list into a GO or pathway enrichment test, and narrate whichever pathway comes out on top as if it had been your hypothesis all along. Each step looks clean. The chain turns a pattern found after the fact into a prediction. And if your starting list already came from a circular selection, the enrichment inherits all of that bias.

Batch effects, a pre-painted target

Sometimes the holes really do cluster, just not for the reason you were hoping. If your cells or samples separate by run, reagent lot, plate, date or sequencing center, your unsupervised analysis will happily split them, and your marker test will happily find genes. You then annotate a "biological state" that actually tracks a technical batch, and build a story on a confounder. Leek and colleagues documented how widespread these batch effects are in high-throughput data and how readily they lead to wrong conclusions, especially when the batch is correlated with the variable of interest.⁴ The target was painted on the wall before your first shot.

Why bioinformatics is an ideal barn wall

Three features enlarge the wall, and they are well catalogued in the single-cell methods reviews.⁵

• Massive multiplicity. Twenty thousand genes times thousands of cells or samples makes an enormous surface for chance clusters to form on.
• No groups defined in advance. Cell types, subtypes, associated variants, that is exactly what you are trying to discover. So the temptation to define and test on the same data is structural.
• Flexible, opaque pipelines. Number of variable genes, number of components, resolution, distance metric, thresholds: that is a pile of knobs you can keep nudging until the picture "makes sense." The multi-magazine barn, quantified.

When the run fails: the sharpshooter at the bench

The fallacy does not wait for the analysis stage. It shows up the moment a run fails, and it may be sneakier there, because no p-value flashes on the wall to warn you.

A failed single-cell run hands you a barn wall of anomalies all at once: low recovery, high mitochondrial fraction, ambient RNA, doublets, an odd saturation curve, a library that migrates strangely on the trace. It is very tempting to spot the single most striking anomaly, paint the causal target around it, and announce "that is why it failed."

What makes this the Texas sharpshooter, and not just a hunch gone wrong? Three ingredients.

Many noisy candidates. A single-cell run has dozens of QC metrics and protocol steps. With that many variables, something will always look off, even in a run that failed for a completely unrelated reason.

No target fixed in advance. If you never defined what "abnormal" means for your system before looking, your expected ranges, your reference runs, then any striking value becomes the cause in hindsight. You see 78% viability and conclude that sank the run. But maybe 78% is normal for your prep, and the real problem sat somewhere else entirely.

The misses go uncounted. The Texan counts only the clustered holes. At the bench, that means scrutinizing the failed run on its own, without checking whether your suspected cause is also sitting in your successful runs. Skip that comparison and any deviation looks like an explanation.

A few concrete shapes it takes:

Circular diagnosis. You form a hypothesis from the failed dataset, then "confirm" it by inspecting more features of that same dataset you already know is abnormal. That is double dipping in a lab coat. You cannot confirm a cause by examining the corpse again. You need a fresh, controlled experiment.

The N=1 leap. One failed run, one salient difference from the last successful one ("we switched enzyme lots"), and there is your causal story. But ten things changed between those two runs. You painted the target around the lot number.

Crediting the fix. You change five parameters at once, the next run works, and you credit the change you already believed in. That one slides toward post hoc ergo propter hoc, though it belongs to the same family: a target drawn after the fact around your preferred intervention.

The bench antidotes are the analysis ones, translated:

• Define your expected QC ranges before you look, from your own historical runs.
• Compare the failed run against your successful ones, metric by metric. This is the control that is almost always missing.
• Treat inspection of the failed dataset as hypothesis-generating, then test the suspected cause with a controlled experiment, ideally changing one factor at a time.
• List the plausible causes up front, a small failure tree, instead of jumping on the first striking one.
• Distrust the single run.

Plenty of post-mortems are perfectly sound reasoning, so do not go paranoid. This bias is specifically about picking one signal out of many, with no target set in advance, while ignoring the misses. When you cling to an explanation mainly because you like it, that leans toward confirmation bias. When you fabricate the hypothesis after seeing the result and present it as if you called it, that is HARKing. These cousins overlap in a single debugging session, and I would rather name them than fold them into one word.

How to stop being the sharpshooter

Good news, the problem is well studied and the fixes exist. They all come back to one idea: fix the target before you shoot.

Treat clustering, and exploration in general, as a hypothesis generator. Structure that emerges hands you targets to aim at. You still have to fire a fresh, independent volley. Concretely: does your cluster reappear, and do its markers hold up, in an independent dataset or cohort where the groups are defined by criteria fixed in advance?

When you genuinely must select and test on the same data, use methods built for it. Three peer-reviewed options:

• Count splitting (Neufeld et al.): split each gene's UMI count into two independent parts under a Poisson assumption, cluster on one, test on the other. Plain sample splitting fails in this specific case, and count splitting restores real Type I error control.⁶
• Selective inference for clustering (Gao, Bien and Witten): a test that explicitly conditions on the fact that the groups came out of clustering, so the p-value means what it claims.¹
• Significance analysis for clustering (Grabski, Street and Irizarry): a hypothesis test folded into the clustering itself, which asks whether two clusters form statistically distinct populations instead of assuming it.³

Watch your technical confounders. Do your clusters track the donor, the run, the plate? Check it before you believe them, and integrate or correct accordingly.

Be upfront about the resolution knob. Report robustness across several resolutions, and do not let "it split into something interesting" become your stopping rule.

Count the multiplicity, including the hidden multiplicity of your pipeline choices. The number of analytical paths available weighs as much as the number of tests.

Report what you tried, not only what worked.

Put it into practice

Principles are easy to nod along to and easy to forget at 2am before a deadline. Here is the concrete version, pitfall by pitfall.

You cluster, then test for markers. Reach for countsplit, the R package from Neufeld and colleagues (install.packages("countsplit")). It splits each gene's counts into independent folds under a Poisson or negative binomial model, so you can cluster on one fold and run differential expression on the other. The authors ship tutorials that plug it straight into Seurat, scran and Monocle3, so the retrofit is small. When you want a test that instead conditions on the clustering itself, the selective-inference approach of Gao, Bien and Witten is the methodological reference.

You build a classifier or a gene signature. Put every data-dependent step, feature selection included, inside the cross-validation loop. In Python that is the scikit-learn pattern of wrapping your selector or a GridSearchCV inside an outer cross_val_score, so nothing from the test fold ever touches selection. In R, tidymodels and caret support the same nested structure. If your reported accuracy drops once you do this, the old number was the mirage.

You report a top hit from a scan (GWAS, DE, screens). Expect the winner's curse on the effect size, and check the magnitude in a genuinely independent cohort before you trust it. For the raw multiplicity, apply Benjamini-Hochberg (p.adjust(method = "BH") in R, multipletests from statsmodels in Python), and count the comparisons you ran informally while exploring, since those inflate things too.

You suspect batch structure. First ask whether batch is confounded with your biological variable. If a batch lines up perfectly with a condition, no software rescues that design, and the right move is to say so and, where possible, redesign or re-collect. When batch and biology are separable, correct with ComBat or ComBat-seq (the sva package) or limma::removeBatchEffect for bulk data, and Harmony, Seurat integration or scVI (scvi-tools) for single-cell, then confirm your clusters no longer track the batch.

You are doing a failed-run post-mortem. Before you open the failed dataset, write down the QC ranges you expect from past successful runs. Then compare the failed run against those runs metric by metric, treat whatever stands out as a hypothesis, and test it by changing one factor at a time.

A short list to run through before you hit publish:

• Did you define your groups, thresholds and endpoints before looking at the results, or after?
• Is any number in your figures computed on the same data that chose what to compute it on?
• Would your main finding survive in an independent dataset where the analysis choices are locked in advance?
• Can you say how many models, resolutions and thresholds you tried before this one?
• Is your key effect corrected for selection (winner's curse, multiple testing)?
• Have you ruled out batch as the real driver?

Answer those cleanly and the target was on the wall before you fired.

Cousins worth keeping apart

Several neighboring biases overlap here, and I would rather name them one by one than pile everything under a single label. Double dipping is selecting and testing on the same data. The winner's curse is the overestimation of effects kept by thresholding. Data leakage is any information from the test set that seeps into training. HARKing is stating your hypothesis after seeing the result while presenting it as prior. The garden of forking paths is the crowd of defensible analytical choices that, taken together, inflate your false positives. The Texas sharpshooter is the thread linking them: a target drawn after the fact around a pattern born of chance.

The one-sentence version

If the analysis that creates your populations, your signatures or your variants is also the one that validates them, be suspicious. You may have just painted a bullseye around your bullet holes, and the p-value on the wall is the paint.