Biomedical Self-Supervised Learning using Sequential and Action-Conditioned Prediction: A Proof of Concept in Histopathology

Self-supervised foundation models for biomedical imaging are typically trained with non-sequential objectives that encourage invariance between augmented crops or predict missing content within a limited spatial resolution or range. This paradigm can be limiting for biomedical images, where diagnostically relevant
structure often depends on relationships across widely different spatial extents and resolutions. We investigate sequential, action-conditioned prediction as an alternative self-supervised pretraining objective for biomedical imaging. As a proof of concept, we adapt seq-JEPA, originally developed for natural images, to 256 × 256 px histopathology images by extending its action space with zoom, and introduce seq-JEPA-dense, which predicts the next representation for every step in the sequence. Under the same per-crop augmentation distribution and with one run per method, seq-JEPA-dense improved tissue classification accuracy in the aggregated linear-probe evaluation from 88.3% to 91.9%,
narrowing the gap to SimCLR, which achieved 93.3%. Both sequential variants recover a held-out crop from the preceding crops and the relative action leading to it at roughly twice the chance rate within the same image, indicating that the prediction depends on the action; whether position, zoom, or both drive this
... mehr
remains to be isolated. These results provide proof-of-concept evidence that sequential, action-conditioned self-supervision can be adapted to histopathology by extending its action space with zoom, yielding representations that approach a contrastive baseline, a first step toward its investigation as a self-supervised
approach for extended biomedical images such as whole-slide images.