--- library_name: transformers license: "mit" tags: - protein-language-model - fastplms --- # FastESMFold ## Model overview `Synthyra/FastESMFold` packages the `facebook/esmfold_v1` checkpoint with the FastPLMs runtime for Hugging Face Transformers. It accepts raw amino-acid sequences through folding helpers, or prepared residue tensors. The repository uses the standard Transformers loading interface with `trust_remote_code=True`. See Technical details for each registered class and whether its weights come from the checkpoint. The sequence- and token-classification classes reuse the pretrained backbone, but their task heads are newly initialized. Fine-tune those heads before interpreting their logits as predictions. ## Install and platform requirements Install the direct dependencies published with this model: ```bash python -m pip install -r \ "https://huggingface.co/Synthyra/FastESMFold/resolve/main/requirements.txt" ``` The FastPLMs implementation itself is embedded in the model repository. Transformers loads it through `trust_remote_code=True`. This model requires Python 3.11-3.14, PyTorch 2.13, and Transformers 5.13. The artifact requirements include the structure dependencies. The release contract requires a CUDA device. The current validated target is the exact NVIDIA GH200 on Linux aarch64. Linux x86-64, CPU-only, Windows, and macOS structure runs are not release evidence. The Hub quick start needs network access for the first download. For an air-gapped run, build the manifest-pinned local artifact first and use the offline example. ## Quick start ```python from transformers import AutoModel model_id = "Synthyra/FastESMFold" model = AutoModel.from_pretrained( model_id, trust_remote_code=True, attn_implementation="sdpa", ).eval() ``` For offline validation, replace `model_id` with the manifest-built `dist/hub/FastESMFold` path. Pass `local_files_only=True`. ## Attention backends The quick start uses `sdpa`. Available backends are `eager`, `sdpa`, `flex_attention`. Requesting an unavailable backend raises instead of silently changing implementation. `output_attentions=True` can use the documented one-call eager fallback to materialize attention tensors. The configured backend does not change. ## Downstream prediction The sequence and token prediction AutoClasses use the checkpoint backbone and create a new, untrained `classifier`. Sequence labels have shape `(b,)`. Residue labels have shape `(b, l)` and use `-100` outside biological positions. The folding trunk is skipped. The classifier uses the checkpoint's learned pLM state mixture and projection, followed by one trainable transformer probe. ```python import torch from transformers import ( AutoModelForSequenceClassification, AutoModelForTokenClassification, ) model_id = "Synthyra/FastESMFold" sequence_model = AutoModelForSequenceClassification.from_pretrained( model_id, num_labels=2, trust_remote_code=True ).eval() token_model = AutoModelForTokenClassification.from_pretrained( model_id, num_labels=3, trust_remote_code=True ).eval() sequences = ["MSTNPKPQRKTKRNT", "MKTIIALSYIFCLVFA"] batch = sequence_model.prepare_classifier_inputs(sequences) biological = batch["attention_mask"].bool() sequence_labels = torch.zeros(len(sequences), dtype=torch.long) token_labels = torch.full_like(batch["input_ids"], -100) token_labels[biological] = 0 with torch.inference_mode(): sequence_output = sequence_model(**batch, labels=sequence_labels) token_output = token_model(**batch, labels=token_labels) print(sequence_output.logits.shape) # (b, 2) print(token_output.logits.shape) # (b, l, 3) ``` ## PEFT fine-tuning Install the training dependencies. Then attach LoRA to the loaded checkpoint: ```bash python -m pip install "datasets>=4.8,<5" "peft>=0.19,<0.20" ``` ```python from peft import LoraConfig, TaskType, get_peft_model peft_model = get_peft_model( sequence_model, LoraConfig( task_type=TaskType.SEQ_CLS, r=8, lora_alpha=16, target_modules="all-linear", modules_to_save=["classifier"], ), ) ``` This checkpoint advertises a classification head. Save the separately trained `classifier` with the adapter. All FastPLMs checkpoints follow the Transformers `PreTrainedModel` contract and can use PEFT. The ESM2-specific shipped CLI is an example, not a support boundary. Record the target modules, base revision, data identity, and trainable parameter scope. ## Protein structure prediction ESMFold accepts a raw sequence and returns structure tensors and confidence: ```python import torch model = model.cuda().eval() with torch.inference_mode(): output = model.infer( "MKTLLILAVVAAALA", num_recycles=4, ) print(output["mean_plddt"]) summary = model.fold_protein( "MKTLLILAVVAAALA", return_pdb_string=True, ) with open("prediction.pdb", "w", encoding="utf-8") as handle: handle.write(summary["pdb_string"]) print(summary["plddt"], summary["ptm"]) ``` FastPLMs does not expose ProteinTTT for ESMFold. The pinned folding checkpoint has no trained masked-language-model head for this objective. `ttt()` and TTT folding requests raise. ## Technical details - Inputs: Raw amino-acid sequences through folding helpers, or prepared residue tensors - Transformers classes: `AutoConfig`, `AutoModel`, `AutoModelForSequenceClassification`, `AutoModelForTokenClassification` - Checkpoint weights: `AutoConfig` = `FastPLMs extension`, `AutoModel` = `pretrained`, `AutoModelForSequenceClassification` = `base weights + untrained task head`, `AutoModelForTokenClassification` = `base weights + untrained task head` - Attention backends: `eager`, `sdpa`, `flex_attention` - Precision: `default` - BF16 execution: `fp32_parameters_autocast` - Generation contract: `not_applicable` - Dependencies: `core + structure` - Weight publication allowed: `true` - Weight license status: `resolved` - Redistributable: `true` - Complete weight publication required: `false` ## Validation and provenance FastPLMs pins the checkpoint, upstream source revisions, state transformation, and required files in `models.toml`. Built artifacts record exact source identities and conversion details in `source-record.json`. - FastPLMs checkpoint: `Synthyra/FastESMFold` - Runtime revision: recorded separately in the built artifact and published commit - Runtime source identities: recorded in `source-record.json` - Official checkpoint: `facebook/esmfold_v1` - Artifact source: `fast` - State transform: `esmfold_meta_to_fastplms_v1` - Pinned upstreams: `fair-esm`, `openfold` - Release tiers: `check`, `compliance`, `structure`, `feature`, `artifact`, `benchmark` - Unresolved required file identities: `0` Release validation includes the `compliance` tier. Its evidence identifies the checkpoint, backend, dtype, hardware, inputs, and reference revision. Declared tiers compare configuration, tokenizer behavior, state, and representative inference with the pinned reference. A nonzero unresolved count blocks release. Metadata alone does not show that a build passed, that a backend is faster, or that an output is biologically valid. ## License Checkpoint terms: MIT. The Hub model-card identifier is `mit`. The local artifact contains applicable source licenses, notices, attribution, and conversion records. Review them before use.