API#
Import as:
import mantispy as mt
Every function that modifies an object takes copy. With copy=False (the default) it
mutates in place and returns None; with copy=True it returns a modified copy and
leaves the input alone. Each call records its arguments under
uns["mantispy"]["params"][<function name>], so a finished object says how it was made.
The Stores column in the tables below lists the keys each function writes.
Reading and writing#
|
Read profiles into an AnnData of observations by features. |
|
Read one plate of images, segmentations and measurements into a |
|
Read JUMP plate profiles, optionally joining the annotation. |
|
Read a mantispy h5ad or zarr store, checking its schema version. |
|
Validate and write |
|
Check |
io.read_profiles reads the output of a profiling pipeline, and the input type decides how. CSV, TSV or parquet
files are stacked, a CellProfiler ExportToSpreadsheet directory is joined across its objects, and a directory of
CytoTable parquet parts is read as single cells. io.read_plate reads the images and segmentations
behind the profiles into SpatialData, from a Cell Painting Gallery source or an ExportForSpatialData plate
folder, and needs the spatial extra:
pip install 'mantispy[spatial]'
Function |
Stores |
|---|---|
|
|
|
returns |
|
as |
|
validates first, then writes h5ad (or zarr for a |
io.validate returns a report rather than raising, unless raise_on_error=True:
- class mantispy._core.schema.ValidationReport(errors=<factory>, warnings=<factory>)[source]#
Result of
validate(). Truthy when there are no errors.
Preprocessing#
|
Mark negative (and optionally positive) controls. |
|
Join the JUMP annotation onto profiles read from the Cell Painting Gallery. |
|
Resolve which |
|
Compute per-cell and per-feature QC metrics. |
|
Drop cells that fail QC or sit in under-populated wells. |
|
Drop all-NaN, low-variance and blocklisted features. |
|
Normalize features within groups, optionally fitting on reference rows only. |
|
Flag the features worth keeping. |
|
Return a new object holding only the features flagged by |
|
Flag outlying cells. |
|
Flag low-quality images and broadcast the verdict onto their cells. |
|
Drop every cell belonging to an image that failed |
|
Flag wells with too few cells, too much missing data, or unstable controls. |
|
Rename features to |
|
Whiten profiles with a transform fitted on the reference rows. |
|
Remove row and column position effects, per plate and per feature. |
|
Regress confounders out of every feature, within groups. |
|
Correct an embedding for batch with Harmony. |
|
Replace every feature by the normal quantile of its rank. |
|
Return at most |
|
Keep features whose values depend on the group, monotonically or otherwise. |
|
Score each feature by how consistently replicates of a perturbation agree on it. |
|
Test each feature for dependence on the batch, after whatever correction was applied. |
Function |
Stores |
|---|---|
|
|
|
|
|
|
|
subsets |
|
subsets |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
returns a new object holding the sampled rows |
|
|
|
|
|
|
pp.normalize also writes var["degenerate_scale"], flagging features with no spread in
some group. Those are divided by epsilon rather than by zero and come back at ~1e17;
drop them before computing anything from distances.
Tools#
|
Aggregate |
|
Mean average precision per group, with a permutation null. |
|
Store pairwise profile similarity in |
|
Median replicate correlation against a non-replicate null. |
|
Similarity of each replicate to its group, z-scored against its similarity to the controls. |
|
One profile per group, weighting replicates by how well they agree. |
|
Per-feature effect size of each group against the reference. |
|
Wasserstein-1 distance per feature between each group and the reference. |
|
Moderated t-test per feature, per group, with wells as the replicates. |
|
Collapse a differential table into a perturbation-by-feature-family matrix. |
|
Call hits by testing each group's distance from the controls. |
|
Energy distance between each group and the controls, or between every pair. |
|
Test whether each perturbation's effect reproduces across settings. |
|
Test whether each compound's response grows with concentration. |
|
Leave-one-out nearest-neighbor mechanism assignment. |
|
Test which mechanisms are over-represented among each profile's nearest neighbors. |
|
Build a decoupler network from the parsed feature annotation. |
|
Score every profile against every feature set. |
|
Rank features by how well they separate each group, with the annotation attached. |
|
Rank feature sets by how far each group's score sits from the rest. |
|
Fraction of each well's cells in each cluster, as a well-level object. |
|
Test each perturbation against the controls within each cluster. |
|
Assign G1, S or G2M from integrated DNA intensity. |
|
Mean distance to the |
|
Score how much a group's signature improves with each additional replicate. |
|
Flag perturbations that both lost cells and moved away from the controls. |
|
Fetch a gene-set network, or read one from a GMT file. |
|
Score how similar the profiles of each gene set's genes are. |
|
Test which gene sets are over-represented among the hits. |
Function |
Stores |
|---|---|
|
returns a new object: |
|
|
|
|
|
|
|
|
|
returns a new object at |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
returns a decoupler network; stores nothing |
|
|
|
|
|
|
|
returns a new object: wells by clusters; |
|
|
|
|
|
|
|
|
|
|
|
returns a gene-set network; stores nothing |
|
|
|
|
tl.map has four modes, named after the questions the field asks. "activity" matches the copairs
reference implementation (0.9267 against 0.9267 over 301 JUMP compounds):
mode |
question |
needs |
|---|---|---|
|
is this perturbation distinguishable from the negative controls? |
controls |
|
do perturbations sharing an annotation look alike, against those that do not? |
|
|
do a perturbation’s replicates retrieve each other against everything else? |
— |
|
the same, counting only replicates from a different plate |
|
Metrics#
|
How well separated the biological labels are, rescaled to |
|
How well mixed the batches are within each biological label. |
|
Median LISI over rows. |
|
Variance-weighted R^2 of the principal components on |
|
|
|
Run every metric for every representation and stack the results. |
|
Check whether differential testing is calibrated on this screen. |
Metrics do not modify the object. Each returns a tidy frame with metric, representation,
key and value, so results from several calls stack. evaluate_correction adds a better
column giving the direction of improvement for each metric, because batch mixing and
biological separation trade off against each other and neither is meaningful alone.
Accessors#
|
Names of the features matching every filter given. |
|
Wide, pycytominer-shaped frame: |
|
Boolean mask of control rows. |
|
Return values for observations in adata. |
|
Return values for observations in adata. |
Accessors do not modify the object. get.to_dataframe returns the flat table that
pycytominer and similar tools expect.
Plotting#
|
Well-grid heatmap of |
|
Distribution of cells per well, split by |
|
Per-feature distributions, before and after normalization when |
|
Fraction of missing values per feature, per plate. |
|
Two-by-two summary of the QC metrics |
|
Row and column medians per plate, for spotting plate position artifacts. |
|
Image quality score per image, with the flagged images marked. |
|
Control wells projected onto principal components fitted on the controls alone. |
|
Outlier score distribution, and the flagged fraction per plate. |
|
Correlation heatmap with features ordered by their annotation. |
|
How many features each group contributes, split by channel. |
|
Heatmap of perturbations by feature families. |
|
Mean average precision against significance, with the strongest groups labeled. |
|
Observed replicate correlation against each group's permutation threshold. |
|
Distance from the controls against significance, with the most distant groups labeled. |
|
The largest effects for one group, colored by feature family. |
|
Effect against significance, per feature, for one group. |
|
One compound's response against dose, with the fitted curve when there is one. |
|
The confusion matrix of |
|
Which mechanisms one profile's neighborhood is enriched for. |
|
The group-by-group distance matrix, ordered so related groups sit together. |
|
Settings against settings: which plates, batches or laboratories agree with each other. |
|
Agreement per perturbation, ranked, with the ones that reproduce colored. |
|
Mean enrichment score per group per feature set. |
|
Stacked bars of cell-state fractions, averaged within each group. |
|
Log DNA intensity per group, colored by assigned phase. |
|
Local cell density against a feature, per group. |
|
Cluster by group heatmap of significance, so an effect in one cell state stands out. |
|
The saturation curve with its spread across draws. |
|
Distance from the controls against viability, with the suspect groups marked. |
|
Coherence per gene set, the significant ones marked. |
|
R^2 of each principal component on each covariate. |
|
Grouped bars of an |
|
Profile-by-profile similarity, ordered by |
There is no dedicated function for a plate map of a per-well flag, which is
mt.pl.plate(adata, color="qc_well_pass"), or for embeddings side by side, which is a loop
over sc.pl.embedding.
Plotting functions return Matplotlib axes and do not modify the object.
Datasets#
|
Generate a synthetic plate with known ground truth. |
|
A small synthetic Cell Painting plate, laid out like one |
|
BBBC021, MCF-7 cells treated with small molecules, the standard mechanism-of-action benchmark. |
|
The Rohban 2017 ORF overexpression screen, with the genes and cell counts that BBBC021 lacks. |
|
Kinase inhibitors over a dose series, from the JUMP pilot. |
|
JUMP-Target-2, one 384-well plate map run at many sites. |
|
Silver nanoparticles, 180 wells at four sizes and three doses. |
|
An Amish cohort, 468 wells varying seeding density and timepoint. |
|
Alternative dyes, 3,455 wells across eight channels. |
|
The assembled JUMP CRISPR arm, 51,185 wells of knockouts. |
|
LUAD alleles, 6,144 wells of wild-type against mutant. |
|
MIAMI, 2,775 wells of compounds in U2OS cells. |
|
Astrocytes and neurons, 1,691 wells imaged at 20x and 63x. |
|
OASIS pilot, 4,604 wells in U2OS and HepaRG. |
|
Pooled rare variants, 290 barcodes in a pooled screen. |
By default io.read_profiles keeps features from the three compartments (Cells, Cytoplasm, Nuclei), matching
pycytominer.infer_cp_features. Whole-field Image_ measurements are excluded (a JUMP plate has 1077 of them
against 3634 per-cell features); objects=None keeps them.
synthetic_plate and blobs are generated locally. synthetic_plate is a single-cell profile table with injected
artifacts for quality control to find; blobs is a small SpatialData plate of images, labels and tables. The other
datasets download once, checked against a pinned sha256, into mt.settings.cache_dir (set MANTISPY_CACHE_DIR to
change it). Four of them carry the annotations the analysis functions need:
dataset |
download |
perturbations |
carries |
|---|---|---|---|
|
~10 MB |
39 compounds |
MOA labels, the classic retrieval benchmark |
|
~27 MB |
194 overexpressed genes |
cell counts, ~10 replicates per gene |
|
~71 MB |
15 kinase inhibitors x 7 doses |
cell counts, MOA labels, 32-64 replicates |
|
~40 MB |
302 compounds, one shared plate map |
the same plate run at two sites, so any difference between them is technical |
The other nine are further gallery accessions, normalized and feature-selected by their authors and read with the
io.read_profiles defaults. Use them to run a method across a range of screens.
Check anything tuned on one dataset against the others. Cutoffs that looked universal on BBBC021 turned out to be
dataset-dependent on rohban and pki.
Settings#
Allows users to customize settings for the |
|
|
Provides local override via keyword arguments as a context manager. |
|
Reset passed settings to their default values. |
Both settings, verbosity and cache_dir, are also read from MANTISPY_VERBOSITY and MANTISPY_CACHE_DIR, and
with mt.settings.override(verbosity=2): changes one for a block.
Relative to scmorph#
scmorph is the other AnnData-based morphological profiling package. mantispy covers its
functionality with two exceptions:
scmorph |
here |
|---|---|
reading CellProfiler CSV output |
|
reading a CellProfiler SQLite database |
not yet (planned) |
quality control, batch correction, aggregation |
|
feature selection |
|
trajectory inference ( |
out of scope: a trajectory through morphology space needs an ordering the assay rarely justifies, and scanpy’s |
Hit calling, effect sizes, dose response, mechanism retrieval, feature-set and pathway enrichment, cell-state composition, replicate power and cytotoxicity have no counterpart in scmorph.
Not reimplemented here#
Dimensionality reduction, neighborhood graphs, embeddings and clustering come from scanpy. Call them directly on the same object:
import scanpy as sc
sc.pp.pca(wells, n_comps=50)
sc.pp.neighbors(wells)
sc.tl.umap(wells)
Harmony is wrapped as pp.harmony. It is the last step of the JUMP consortium’s recipe
[CCG+24] and the best performer in Arevalo et al. [ASE+24]. It needs the
optional extra:
pip install 'mantispy[harmony]'
It corrects an embedding rather than the features, so run sc.pp.pca first. The other
corrections are pp.sphere, pp.correct_plate_position and pp.regress_out.
The data contract#
X is float32, one row per profile. obs carries Metadata_ columns identifying
where each profile came from; var carries the parsed feature annotation
(object, feature_group, feature, channel, scale, angle, gray_levels,
radial_bin, params, is_feature). uns["mantispy"] holds the schema version, the
resolution, the channel vocabulary and provenance.
Resolution ("cell", "well", "perturbation") is advisory. It sets which identifier
columns io.validate requires, and functions warn instead of raising when they get a
resolution they do not expect.
The machine-readable contract is published as spec/schema-1.0.json, alongside the
earlier spec/schema-0.1.json.
Seeing what mantispy is doing#
mantispy logs what it drops, skips and shrinks through the standard library’s logging.
Warnings print by default; raise the verbosity to see the rest:
import mantispy as mt
mt.settings.verbosity = 2 # 0 errors, 1 warnings (default), 2 info, 3 debug
Set it to 2 when you first run a new screen. Several functions discard features or wells, and they log it at that level.
Working backed#
mt.io.read(path, backed="r") leaves X on disk. obs and var stay in memory, so
metadata, QC flags and feature selection work unchanged. Functions that rewrite X need
copy=True and raise a ValueError saying so when it is missing.
operation |
backed |
|---|---|
|
yes (they write |
|
yes (they return new objects or tables) |
|
with |
writing |
no; the file is opened read-only |
Grouped reductions read one group at a time when the matrix is on disk, and the tests assert that the results are identical to the in-memory path. For a per-plate median on 100 000 cells x 500 features, backed mode peaks at 44 MB against a 200 MB resident matrix in memory, and takes 3.5 s against 0.8 s. That is about four times the runtime for four times less memory, so use it only on data that does not fit in memory.
Performance#
benchmarks/ holds a suite that is not part of the test run; run it with pytest benchmarks -s.
Measured on a laptop, 100 000 cells by 500 features (X is 200 MB):
operation |
time |
peak |
|---|---|---|
|
0.86 s |
200 MB |
|
2.06 s |
232 MB |
|
0.41 s |
458 MB |
|
0.39 s |
78 MB |
|
0.48 s |
227 MB |
|
0.21 s |
7 MB |
|
65 s |
2465 MB |
At 500 000 cells the preprocessing scales linearly: normalize 12.7 s, aggregate 2.0 s.
tl.map is by far the most expensive step, in both time and memory.
Stability#
The schema is frozen at 1.0. The names in spec/schema-1.0.json (the required and
reserved obs columns, the var annotation, the uns["mantispy"] keys and the result
tables) are stable, as are the public signatures in the tables above.
Anything removed gets a
DeprecationWarningfor two minor releases first.mt.io.readmigrates a file written against an older schema on read. A schema version this build does not know raises an error._coreand other_-prefixed modules are private.
io.read_plate and ds.blobs bring images and segmentations in as SpatialData on top of this contract, without
changing it.