wellesley-easel/randall-segmentation-model

Randall segmentation model

This private research checkpoint segments medieval manuscript pages into eight semantic classes. It is the epoch_119 checkpoint from the multiclass_vitl_depthwise_final experiment. Checkpoint numbers are zero based, so it was saved after training epoch 120 and is the latest persisted checkpoint in that experiment directory.

Important: base model required

model.pth is not a standalone SAM checkpoint. It contains the trained rank-8 DoRA parameters, four depthwise self-prompt-generator branches, prompt encoder, and eight-class mask decoder. Inference also requires Meta's original SAM ViT-L checkpoint, sam_vit_l_0b3195.pth, obtained separately under its own terms.

The model implementation is the MapSAM extension at source revision c544298. It is not directly loadable with the Transformers AutoModel classes.

Classes

ID	Label
0	unlabeled
1	line filler
2	marginal images
3	painted initials
4	miniatures
5	borders
6	pen flourishing
7	pen flourished texts

Evaluation

The checkpoint was evaluated on 154 held-out manuscript pages after retiling 764 model-input tiles to page resolution.

Metric	Value
Overall pixel accuracy	0.953899
Foreground pixel accuracy	0.702386
Foreground mean IoU (classes 1–7)	0.552400
Marginal-images IoU	0.460426
Marginal-images Dice	0.630537

Class	IoU	Dice
unlabeled	0.967716	0.983593
line filler	0.665873	0.799428
marginal images	0.460426	0.630537
painted initials	0.525954	0.689345
miniatures	0.618410	0.764220
borders	0.553674	0.712729
pen flourishing	0.507451	0.673257
pen flourished texts	0.535014	0.697080

Full counts, precision, recall, and the confusion matrix are in evaluation/metrics.json.

Loading and inference

Clone the source revision above, install its dependencies, download the base SAM ViT-L checkpoint, and download this repository's files. From the source checkout, inference follows this pattern:

python run_inference.py \
  --input_dir INPUT_IMAGES \
  --output_dir OUTPUT_SEGMENTATIONS \
  --dora_ckpt model.pth \
  --ckpt sam_vit_l_0b3195.pth \
  --config config.txt \
  --id2label id2label.json

config.txt contains only inference-relevant settings and intentionally omits the machine-local paths recorded by the training job. config.json provides the same architecture metadata in a structured form.

Training summary

• Backbone: SAM ViT-L
• Input size: 1024 × 1024
• Output resolution: 256 × 256 before page-space resizing
• DoRA rank: 8, applied to image-encoder query and value projections
• Self-prompt generator: depthwise variant
• Optimizer: AdamW
• Base learning rate: 0.0001 with 750-iteration warmup
• Batch size: 6
• Training data: 340 manuscript pages converted to 1,925 tiles
• Evaluation data: 154 pages converted to 764 tiles

The test split was evaluated repeatedly during training, so the reported test metrics should be treated as descriptive rather than as an unbiased estimate from a never-observed final test set.

Limitations and intended use

This model was trained on a specialized collection of medieval manuscript images. Performance may degrade for other repositories, imaging conditions, layouts, periods, or visual traditions. Small, ambiguous, or overlapping decorations can be confused, and predictions require human review for scholarly use. It is not intended for consequential automated decisions.

This repository is a private review draft. No public redistribution license has been assigned to these fine-tuned weights. The separately obtained SAM base checkpoint remains governed by its own license.