dolphinteam/OpenWhistle-BioClust

OpenWhistle BioClust

dolphinteam/OpenWhistle-BioClust provides a Wav2Vec2 audio backbone pretrained from scratch on bottlenose dolphin whistle recordings using native Fairseq on three NVIDIA L40S GPUs at Bioclust.

It follows the repository layout and frontend geometry of OpenWhistle-Wav2Vec2.0. The weights come from the completed Bioclust run at 400,000 optimizer updates.

Model Details

• Model type: wav2vec2
• Transformers architecture: Wav2Vec2ForPreTraining
• Hidden size: 768; Transformer layers: 12; attention heads: 8
• Seven convolutional layers, with strides [15, 2, 2, 2, 2, 2, 2]
• Total stride: 960 audio samples, approximately 21.77 ms at 44,100 Hz
• Sampling rate: 44,100 Hz; mono audio
• Weights: model.safetensors, exported in float32

The export maps every learned Fairseq tensor strictly and checks encoder embeddings against the native checkpoint. Conversion details and the checkpoint SHA-256 are recorded in export_provenance.json.

Training and Evaluation Data

Pretraining uses dolphinteam/OpenWhistle-Pretraining, pinned to revision eea37423c7841b0f99a7c0380f945999c0a2c489.

The manifests contain 28,410 training examples and 3,370 validation examples. Fairseq excludes two training examples shorter than five seconds. Existing 44.1 kHz mono WAVs were reused after checking their headers and metadata; 55 missing or invalid examples were reconstructed from the matching Arrow cache as PCM_24 FLAC files.

Training completed on 12 October 2026 with native Fairseq 0.12.2, PyTorch 2.5.1/CUDA 12.4, FP16, seed 42, 32,000 warmup updates and a peak learning rate of 0.0005. Each GPU has a maximum audio budget of 4,704,000 samples per update, with accumulation 1 and crops capped at 20 seconds. Validation and checkpointing occur every 5,000 updates.

The final native Fairseq validation loss is 0.853 at 400,000 updates. The best observed validation loss is 0.852 at 395,000 updates. This repository publishes the final 400,000-update checkpoint. These self-supervised losses do not measure downstream classification or detection performance.

The OpenWhistle supervised datasets are available for subsequent fine-tuning and evaluation:

• OpenWhistle-Classification-Finetuning
• OpenWhistle-Detection-Finetuning

Intended Use

Use the backbone to extract audio representations or fine-tune a downstream model for bottlenose dolphin whistle classification and detection. This checkpoint has no supervised classification or detection head.

Input Audio

Provide mono waveforms sampled at 44,100 Hz. The feature extractor normalizes the waveform and uses right padding with value 0.0 and return_attention_mask=False.

For comparisons with the unpadded native encoder, process waveforms individually or batch equal-length waveforms. Resample audio recorded at other sampling rates before processing.

Loading

import torch
from transformers import AutoFeatureExtractor, AutoModel

repo_id = "dolphinteam/OpenWhistle-BioClust"
processor = AutoFeatureExtractor.from_pretrained(repo_id)
model = AutoModel.from_pretrained(repo_id).eval()

# waveform is a one-dimensional mono array sampled at 44,100 Hz.
inputs = processor(waveform, sampling_rate=44100, return_tensors="pt")
with torch.no_grad():
    embeddings = model(**inputs).last_hidden_state

To load the quantizer and projection heads as well:

from transformers import Wav2Vec2ForPreTraining
model = Wav2Vec2ForPreTraining.from_pretrained(repo_id)

Transformers pretraining uses its own masking and loss implementation. The original training recipe and metrics come from native Fairseq.

Files

• config.json: model architecture and Transformers configuration
• model.safetensors: converted encoder and pretraining-head weights
• preprocessor_config.json: audio preprocessing settings
• trainer_state.json: native Fairseq metrics reconstructed in the reference Trainer-state layout
• export_provenance.json: checkpoint identity, tensor mapping and embedding parity
• training_provenance.json: dataset revision, training settings and validation history

The native optimizer, scheduler and RNG states are retained in the Bioclust Fairseq checkpoint and are not included in this inference repository. trainer_state.json is a metrics record, not a Transformers training-resume artifact; training FLOPs were not tracked.

Limitations

• Training focuses on bottlenose dolphin whistles; transfer to other species or recording settings requires evaluation.
• Downstream results for this checkpoint have not yet been established.
• Biological interpretations of embeddings require independent validation.

License

No model-weight license has been specified, following the reference repository. Contact the OpenWhistle maintainers for licensing terms.

Paper, collection, and authors

• Paper: OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations
• Hugging Face collection: NeurIPS 2026 OpenWhistle
• This repository: https://huggingface.co/dolphinteam/OpenWhistle-BioClust

Authors: Faadil Mustun, Chiara Semenzin, Roberto Dessi, Pablo Robin Guerrero, Pierre Orhan, Alexis Emanuelli, Emanuele Rossi, Yair Lakretz, Gonzalo de Polavieja, and German Sumbre.

Citation

@misc{mustun2026openwhistle,
  title         = {OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations},
  author        = {Mustun, Faadil and Semenzin, Chiara and Dessi, Roberto and Robin Guerrero, Pablo and Orhan, Pierre and Emanuelli, Alexis and Rossi, Emanuele and Lakretz, Yair and de Polavieja, Gonzalo and Sumbre, German},
  year          = {2026},
  eprint        = {2609.34839},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.34839}
}