Rnafm

RNA-FM: Foundation Model for Non-Coding RNA

Overview

RNA-FM is a BERT-style transformer pretrained on 23 million non-coding RNA (ncRNA) sequences via masked language modeling. It generates contextual nucleotide embeddings that capture RNA structure and function without labeled data.

Model variants:

• RNA-FM — trained on ncRNA sequences (23M); embedding dim = 640; tokenizes individual nucleotides
• mRNA-FM — trained on 45M mRNA coding sequences (CDS); embedding dim = 1280; tokenizes 3-mers (codons)

Capabilities:

• Sequence embeddings — per-token or pooled representations for any downstream task
• Secondary structure prediction — base-pair contact maps (outperforms LinearFold/SPOT-RNA)
• RNA family clustering — zero-shot separation of RNA families in embedding space
• Functional prediction — UTR function, gene expression, RNA-protein binding

API note: RNA-FM's Python API mirrors ESM2 — (model, alphabet) pair, batch_converter, repr_layers. If you know ESM2, RNA-FM will feel familiar.

Installation

pip install rna-fm

Requirements: Python ≥ 3.8, PyTorch ≥ 1.9. GPU with CUDA 11.1+ recommended.

Model weights download automatically on first use (~1.2 GB for RNA-FM, ~957 MB for mRNA-FM).

Model Checkpoints

| Checkpoint | Training data | Embed dim | Tokenization | Best for | |---|---|---|---|---| | rna_fm_t12 | 23M ncRNA sequences | 640 | Single nucleotide | ncRNA, structural RNA, general RNA | | mrna_fm_t12 | 45M mRNA CDS sequences | 1280 | 3-mer (codon) | mRNA analysis, codon usage, translation |

Core Usage

Load model

import fm

# ncRNA model (default)
model, alphabet = fm.pretrained.rna_fm_t12()

# mRNA model
model, alphabet = fm.pretrained.mrna_fm_t12()

model.eval()

Extract embeddings

import torch
import fm

model, alphabet = fm.pretrained.rna_fm_t12()
model.eval()

batch_converter = alphabet.get_batch_converter()

# Input: list of (label, sequence) tuples
sequences = [
    ("rna1", "GGGUGCGAUCAUACCAGCACUAAUGCCCUCCUGGGAAGUCCUCGUGUUGCACCCCU"),
    ("rna2", "AUGUAAGGCCUUGUAACGCUCUAAACUUCCCCCGCGACGUUUUU"),
]

batch_labels, batch_strs, batch_tokens = batch_converter(sequences)

with torch.no_grad():
    results = model(batch_tokens, repr_layers=[12])

# Per-token embeddings from last layer: (batch, seq_len, 640)
token_embeddings = results["representations"][12]

# Per-sequence mean pooling (exclude BOS/EOS/PAD)
padding_idx = alphabet.padding_idx
for i, label in enumerate(batch_labels):
    mask = (batch_tokens[i] != padding_idx).float()
    seq_emb = (token_embeddings[i] * mask.unsqueeze(-1)).sum(0) / mask.sum()
    print(f"{label}: {seq_emb.shape}")  # (640,)

Secondary structure prediction

# CLI
python launch/predict.py \
  --config="pretrained/ss_prediction.yml" \
  --data_path="sequences.fasta" \
  --save_dir="results/"

Output: contact map in CT format (base-pair predictions).

Extract embeddings via CLI

# Mean-pooled embeddings
python launch/predict.py \
  --config="pretrained/extract_embedding.yml" \
  --data_path="sequences.fasta" \
  --save_dir="results/" \
  --save_embeddings \
  --save_embeddings_format="mean"

# Per-token (raw) embeddings
python launch/predict.py \
  --config="pretrained/extract_embedding.yml" \
  --data_path="sequences.fasta" \
  --save_dir="results/" \
  --save_embeddings \
  --save_embeddings_format="raw"

# BOS token (sequence-level)
python launch/predict.py \
  --config="pretrained/extract_embedding.yml" \
  --data_path="sequences.fasta" \
  --save_dir="results/" \
  --save_embeddings \
  --save_embeddings_format="bos"

Key Parameters

| Parameter | Value | Description | |---|---|---| | repr_layers | [12] | Which transformer layers to extract (0–12; 12 = last) | | Max sequence length | 1024 nt | Hard limit; truncate longer sequences | | Embedding dim | 640 (RNA-FM) / 1280 (mRNA-FM) | Output size per token | | need_head_weights | False | Return attention weights (memory-intensive) | | return_contacts | False | Return contact predictions |

Special Tokens

RNA-FM uses ESM-1b-style tokenization:

• `` / BOS prepended at position 0
• `` appended at end
• `` fills shorter sequences in a batch

When extracting per-residue embeddings for downstream use, slice [1:-1] to remove BOS and EOS:

per_residue = token_embeddings[i, 1:-1]  # shape: (seq_len, 640)

For mean pooling, mask out padding tokens using alphabet.padding_idx.

Output

| Format | Shape | Description | |---|---|---| | repr_layers=[12] | (batch, seq_len, 640) | Per-token embeddings (includes BOS/EOS) | | Mean pooled | (batch, 640) | Sequence-level representation | | BOS token | (batch, 640) | CLS-style representation | | logits | (batch, seq_len, vocab_size) | Masked LM prediction logits |

Scripts

• scripts/embed.py — embed FASTA sequences, save as .npy or .h5; see scripts/embed.py --help

Related Models (ml4bio)

The ml4bio lab builds an RNA design ecosystem around RNA-FM:

| Model | Task | Repo | |---|---|---| | RhoFold+ | RNA 3D structure prediction | ml4bio/RhoFold | | RhoDesign | RNA inverse folding (structure → sequence) | ml4bio/RhoDesign | | RiboDiffusion | Diffusion-based RNA inverse folding | ml4bio/RiboDiffusion |

Resources

• GitHub: https://github.com/ml4bio/RNA-FM
• Paper: Chen et al., Nature Methods 2024 — https://doi.org/10.1038/s41592-024-02487-0

References

• references/api-reference.md — full API reference: BatchConverter, model.forward(), embedding formats, secondary structure, mRNA-FM differences

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: naity
• Source: naity/FM4Life
• License: MIT

Install and usage instructions live in the source repository linked above.