In this blog, we examine how the “perturbation effect” can vary depending on the metrics used to define it, and why these differences matter. While these metrics may appear interchangeable, they often capture fundamentally different aspects of the underlying biology. As Perturb-seq datasets continue to grow exponentially, understanding how perturbation effects are measured becomes critical for reliable downstream analysis.

When suppression is not an on-off switch

In 2025, Nadig and colleagues released a CRISPRi perturbation study of more than 2,000 genes on hepatocellular carcinoma cells. They discovered that perturbation effects sometimes are undetectable using normal downstream analysis. This observation is interesting to us because the data scientists at DataXight also recognized inconsistent patterns when looking at perturbation of different target genes on the same cell lines.

Our observation is as follows: we compared the expression of each targeted gene between perturbed cells and non-targeting controls to assess the effectiveness of CRISPRi-mediated knockdown across all perturbations. We measured the level of significance using a standard Wilcoxon rank-sum test with Bonferroni correction. The suppression spans a gradient of effects across 2,160 perturbations, of which 60.2% (1,300 genes) showed a statistically significant decrease in expression, while 39.8% (860 genes) did not.

This variability in distribution has also been acknowledged by the authors (Nadig et al., 2025), suggesting that CRISPRi inference is inconsistent across perturbations. Therefore, if we evaluate the “perturbation effect” through the lens of expression values of the target gene, it should be interpreted as a spectrum rather than just “perturbed” and “controlled”.

Figure 1. Pie chart showing the proportion of target genes with statistically significant downregulation following CRISPRi perturbation (Wilcoxon rank-sum test, Bonferroni-corrected α = 0.05).

What are the alternatives? 

Since knockdown efficiency varies widely across genes, it raises a question: What other ways can we use to look at perturbation effects?

In this investigation, we examined three types of metrics and their relations with each other:

  • Gene expression: the expression of the target gene in perturbed cells.
  • Gene rank: an ordinal ranking of the target gene compared to other genes (1 is the highest)
  • Fold-change: the ratio of target gene expression between the perturbed and control cells.

We performed a simple correlation test between the 3 metrics. We found signs of correlation in gene expression vs gene rank and gene expression vs fold-change. Interestingly, there is no significant correlation in gene rank vs fold change.

Figure 2. Pairwise scatter plots of three perturbation metrics across 2,160 gene knockdowns. (A) Mean perturbed expression vs. log₂ fold-change. (B) Mean perturbed expression vs. gene rank. (C) Gene rank vs. log₂ fold-change. R is correlation coefficient, and p is p-value.

We realized that even though these metrics are both derived from gene expression, each tells a different part of the perturbation story. Different from gene rank, which relies on the relativity of the target gene expression to other genes in the perturbed cells, fold change uses the expression of control cells. So there are two possible explanations, both can be true at the same time:

  • The control baseline varies significantly and does not follow a monotonic relationship with gene expression.
  • The target’s expression changes in a way that is not reflected relative to the rest of the perturbed transcriptome.

Since both metrics are correlated with gene expression but have different implications, we anticipate that using them for downstream analysis will help us see a different picture of the perturbation effect.

From multi-dimensions to two dimensions

Following the observation of alternate metrics, we further investigated their differences in dimensionality reduction, a common downstream visualization to picturize cell states. The gene expression and fold-change data were projected to two dimensions using UMAP. We then applied the same label for both UMAPs using cell clusters calculated from the gene expression data.

Figure 3. Single-cell UMAP projections of 140,497 perturbed cells in normalized expression space and log₂ fold-change space, colored by clusters derived from PCA-based Leiden clustering on normalized expression

The two UMAP embeddings diverge in their organization, indicating that the choice of metric materially alters the observed structure. In particular, the Cluster 3 (21,107 cells) and Cluster 5 (10,265 cells) are clearly separated in the normalized expression UMAP (Panel A)and would be interpreted as distinct cell populations or states (Figure 4). However, in the Fold-change UMAP (Panel B), the same cells, labeled with identical cluster assignments, collapse into a single, overlapping group, suggesting that the cell clusters observed in expression space are not preserved once measurements are reframed to the “relative” expression.

Figure 4. UMAP projections as in Figure 2, with cells from Cluster 3 and Cluster 5 highlighted against the full dataset background.

Notably, 88.7% of perturbations are shared between Cluster 3 and Cluster 5, indicating that they represent nearly identical sets of gene knockdowns. Their apparent distinction in expression space is therefore driven by differences in absolute expression levels rather than perturbation-induced effects. This also indicates the heterogeneity within one perturbation. Without the lens of “relative” expression, these strong signals can be quite misleading.

Seeing Biology Through the Right Lens

When helping our clients look at perturbation data, we carefully selected different metrics and tried to faithfully present the implications. Perturbation is an emerging field, and new methods will be developed to provide us with better and better “lenses” into the perturbation effect. However, before relying on complex models or automated interpretations, it is essential to understand what the underlying measurements actually represent and what they do not.

In this blog, we discussed the following fundamentals:

  • CRISPRi effects are inherently variable, not uniform: Knockdown effects span a spectrum, with many targets showing non-detectable suppression, highlighting that genetic perturbations are neither consistently effective nor directly comparable in magnitude.
  • Metrics Tell Different Stories: Absolute expression, fold-change, and rank capture distinct aspects of the data, show limited agreement, and cannot be assumed to proxy the others. These differences have real implications for how perturbation data is interpreted. Depending on the metric used, the same dataset can lead to different conclusions about gene function, cell states, and biological mechanisms. In practice, this means that researchers may draw incomplete or even misleading insights if they rely on a single analytical lens.
  • Analytical conclusions are metric-dependent: Downstream results, including clustering, similarity, and interpretation of perturbation effects, can shift substantially due to the chosen metric. Findings should therefore be interpreted within the context of how the data were represented, rather than assumed to be metric-invariant.

Data and additional information

Data source: Nadig, A., Replogle, J.M., Pogson, A.N. et al. Transcriptome-wide analysis of differential expression in perturbation atlases. Nat Genet 57, 1228–1237 (2025). https://doi.org/10.1038/s41588-025-02169-3.

Preprocessing: Quality control (mitopercent < 20%, UMI > 1,000), library-size normalization (10,000 counts per cell), log1p transformation, minimum 5 cells per perturbation. Analysis performed in Python using anndata, scanpy, scikit-learn, and matplotlib.