Docs

Research

A deep dive into single-cell RNA sequencing foundation models

Large-scale foundation models, which are pre-trained on massive, un-labeled datasets and subsequently fine-tuned on specific tasks, have recently achieved unparalleled success on a wide array of applications, including in healthcare and biology. In this paper, we explore two foundation models recently developed for single-cell RNA sequencing data, scBERT and scGPT. Focusing on the fine-tuning task of cell type annotation, we explore the relative performance of pre-trained models compared to a simple baseline, L1-regularised logistic regression, including in the few-shot setting. We perform ablation studies to understand whether pre-training improves model performance and to better understand the difficulty of the pre-training task in scBERT. Finally, using scBERT as an example, we demonstrate the potential sensitivity of fine-tuning to hyper-parameter settings and parameter initialisations. Taken together, our results highlight the importance of rigorously testing foundation models against well established baselines, establishing challenging fine-tuning tasks on which to benchmark foundation models, and performing deep introspection into the embeddings learned by the model in order to more effectively harness these models to transform single-cell data analysis. Code is available at https://github.com/clinicalml/sc-foundation-eval.

Details

author(s)
David Sontag
publication date
23 October 2023
source
bioRxiv
related programme
MIT Jameel Clinic
Link to publication
External link ->

Generative AI in the era of 'alternative facts'

|

MIT Open Publishing Services

External data and AI are making each other more valuable

|

Harvard Business Review Press

Removing biases from molecular representations via information maximisation

|

Arxiv

Effective human-AI teams via learned natural language rules and onboarding

|

Arxiv

A deep dive into single-cell RNA sequencing foundation models

|

bioRxiv

Antibiotic identified by AI

|

Nature

LLM-grounded video diffusion models

|

Arxiv

Successful Development of a Natural Language Processing Algorithm for Pancreatic Neoplasms and Associated Histologic Features

|

Pancreas

Leveraging artificial intelligence in the fight against infectious diseases

|

Science

BioAutoMATED: An end-to-end automated machine learning tool for explanation and design of biological sequences

|

Cell Systems

Conformal language modeling

|

Arxiv

Comparison of mammography AI algorithms with a clinical risk model for 5-year breast cancer risk prediction: An observational study

|

Radiological Society of North America

Deep learning-guided discovery of an antibiotic targeting Acinetobacter baumannii

|

Nature

Algorithmic pluralism: A structural approach towards equal opportunity

|

Arxiv

Artificial intelligence and machine learning in lung cancer screening

|

Science Direct

Wide and deep neural networks achieve consistency for classification

|

PNAS

Autocatalytic base editing for RNA-responsive translational control

|

Nature

DiffDock: Diffusion steps, twists and turns for molecular docking

|

Arxiv

Sybil: A Validated Deep Learning Model to Predict Future Lung Cancer Risk From a Single Low-Dose Chest Computed Tomography

|

Journal of Clinical Oncology

Sequential multi-dimensional self-supervised learning for clinical time series

|

Proceedings of Machine Learning Research

Queueing theory: Classical and modern methods

|

Dynamic Ideas

Toward robust mammography-based models for breast cancer risk

|

Science

The age of AI: And our human future

|

Little, Brown and Company

Uniform priors for data-efficient transfer

|

Arxiv

Machine learning under a modern optimisation lens

|

Dynamic Ideas

The marginal value of adaptive gradient methods in machine learning

|

Advances in Neural Information Processing Systems

Efficient graph-based image segmentation

|

International Journal of Computer Vision

We use cookies on our site.