Panoptes

A 3-branch, multi-resolution InceptionResNet CNN that predicts histological subtype, molecular subtype, and gene mutation status directly from H&E-stained whole-slide images — originally built for endometrial carcinoma (Hong et al., Cell Reports Medicine 2021). This page mirrors what the paper states the workflow to be; the code browser lets you read the actual implementation cloned from GitHub, and the runner actually trains and tests it, for real, on a bundled demo slide pair or slides you upload.

Repo cloned master 831fe9d 11.9.1 Legacy Python 3.7 / TF1.13 / Keras 2.3 env installed & runnable

Run it → Browse the code → Read the paper →

What it predicts

Histological subtype

Endometrioid vs. Serous

Molecular subtypes (4)

CNV-H · CNV-L · MSI-H · POLE

Gene mutations

18 genes, one binary model each

ARID1A ATM BRCA2 CTCF CTNNB1 FAT1 FBXW7 FGFR2 JAK1 KRAS MTOR PIK3CA PIK3R1 PPP2R1A PTEN RPL22 TP53 ZFHX3

Architecture variants

Panoptes1 (P1)
InceptionResNetV1-based, 3 resolution branches.

Panoptes2 (P2)
InceptionResNetV2-based, 3 resolution branches.

Panoptes3 (P3)
InceptionResNetV1-based, alternate merge/1×1 conv config.

Panoptes4 (P4)
InceptionResNetV2-based, alternate merge/1×1 conv config.

PC1–PC4 variants add a 4th branch merging patient BMI + age before the final classifier.

Workflow — paper vs. code

1. Tiling

Paper says

Slides opened with OpenSlide (SVS/SCN). Tiles cut at 2.5×, 5×, and 10× equivalent resolution, 299×299 px each; each higher-resolution tile covers one-fourth the area of the next lower one (same center point, 3 concentric fields of view).

Code does

Slicer.py:v_slide() reads region at (level, tile_size), scales offsets by 4**level to align the 3 resolutions, then resizes every tile to 299×299.

2. Background / artifact filter

Paper says

Tiles with more than 40% white background or contaminants excluded.

Code does

Slicer.py:bgcheck() flags pixels that are all-channel >200 (white) or all-channel <50 (dark contaminant) and returns their combined fraction; v_slide() keeps a tile only if that fraction is < 0.8 (i.e. discards at ≥80% background/contaminant, not 40%).

Note: The 0.8 cutoff in the shipped code is looser than the 40% the paper text describes — worth knowing if you replicate their filtering exactly rather than trusting the prose.

3. Stain normalization

Paper says

Vahadane method, via the `staintools` package, against a fixed reference tile.

Code does

Slicer.py:normalization() calls staintools.LuminosityStandardizer then StainNormalizer(method='vahadane').fit(std_img) — std_img is colorstandard.png at repo root.

4. Storage

Paper says

Tiles for each set (train/val/test) written to a single TFRecords file.

Code does

data_input.py builds TFRecords from the tile lists produced by prep.py/Slicer.py.

5. Model

Paper says

3-branch InceptionResNet (one branch per resolution), branches concatenated near the output, then global average pooling and a final dense classifier. 4 variants (Panoptes1-4) differ in InceptionResNet version (V1/V2) and an optional 1×1 conv / clinical-feature (age, BMI) branch.

Code does

PanoptesN.py builds 3 stems/blocks (one per input tile), concatenate(axis=3) at '8x8x2688', GlobalAveragePooling2D, optional demographics Dense(2) branch merged in, final Dense(num_cls) classifier.

6. Training

Paper says

Adam, initial LR 1e-4, dropout keep-rate 0.3, batch 24 (Panoptes) or 64 (baselines). Checkpoint on min validation loss (100 batches every 1000 iters); early stop after 10,000 iters with no improvement, minimum 100,000 iters run.

Code does

cnn.py's INCEPTION class defaults to batch_size=24, dropout=0.3, learning_rate=1e-3, but Main.py's HYPERPARAMS dict overrides learning_rate to 1e-4 at call time — the class default is a red herring; the actual run matches the paper.

7. Prediction tasks

Paper says

Histological subtype (endometrioid vs. serous); 4 molecular subtypes (CNV-H, CNV-L, MSI-H, POLE); 18 gene mutations (ARID1A, ATM, BRCA2, CTCF, CTNNB1, FAT1, FBXW7, FGFR2, JAK1, KRAS, MTOR, PIK3CA, PIK3R1, PPP2R1A, PTEN, RPL22, TP53, ZFHX3).

Code does

Main.py's `feature` CLI arg selects the label column(s); README lists the same 18 mutations, 4 subtypes, histology task.

8. Aggregation & evaluation

Paper says

Per-tile prediction scores averaged per patient; AUROC (bootstrapped 95% CI) computed per-patient; heatmaps built by mapping tile predictions back to slide coordinates.

Code does

accessory.py implements the metrics/aggregation and heatmap plotting.

Cloned from github.com/rhong3/Panoptes into /srv/flask/server_home/panoptes/repo. The stack is TensorFlow 1.13 / Keras 2.3 (2019–2020) — it doesn't install on this machine's main Python 3.12 environment, so it runs in a dedicated conda env (panoptes, Python 3.7) instead. There is no publicly available pretrained checkpoint for this model (the paper's TCGA/CPTAC training data is access-controlled) — the runner trains a real (tiny, non-diagnostic) model from scratch on either a bundled 4-slide demo or slides you upload, then applies it, so every step of the real pipeline actually executes.