See our collection for all versions of OWLv2.

Run OWLv2 with Keras 3: JAX, PyTorch, or TensorFlow

kerasformers/owlv2-base-patch16

Paper: Scaling Open-Vocabulary Object Detection (arXiv:2306.09683) · HF Papers

OWLv2 keeps OWL-ViT's dual-tower skeleton and per-patch detection head, and scales it with self-training on web image-text pairs. It adds an objectness head (a learned is-this-patch-an-object score) and pads images to a square before resizing, which matters for post-processing target sizes.

For more details on the model, please go to Google's original model card.

Pure-Keras 3 conversion of google/owlv2-base-patch16 for kerasformers. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is an open-vocabulary object detection checkpoint (Owlv2Detect): pass free-text prompts at inference time.

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from kerasformers.models.owlv2 import (
    Owlv2Detect,
    Owlv2Processor,
    Owlv2ImageProcessor,
)

model = Owlv2Detect.from_weights("kerasformers/owlv2-base-patch16")
processor = Owlv2Processor.from_weights("kerasformers/owlv2-base-patch16")
image_processor = Owlv2ImageProcessor.from_weights("kerasformers/owlv2-base-patch16")

image = Image.open("your_image.jpg").convert("RGB")
prompts = ["a photo of a mug", "a photo of a knife"]
inputs = processor(text=[prompts], images=image)
output = model(
    {
        "input_ids": inputs["input_ids"],
        "pixel_values": inputs["pixel_values"],
    }
)
results = image_processor.post_process_object_detection(
    output,
    threshold=0.1,
    target_sizes=[(image.height, image.width)],
    text_labels=[prompts],
)[0]
for score, name, box in zip(
    results["scores"], results["text_labels"], results["boxes"]
):
    print(f"{name}: {float(score):.3f} {box}")

Load any OWL-ViT / OWLv2 variant the same way with from_weights("kerasformers/<variant>") (use Owlv2Detect for this repo):

Variant	Hub	Family
`owlvit-base-patch32`	`kerasformers/owlvit-base-patch32`	OWL-ViT
`owlvit-base-patch16`	`kerasformers/owlvit-base-patch16`	OWL-ViT
`owlvit-large-patch14`	`kerasformers/owlvit-large-patch14`	OWL-ViT
`owlv2-base-patch16`	`kerasformers/owlv2-base-patch16`	OWLv2
`owlv2-base-patch16-ensemble`	`kerasformers/owlv2-base-patch16-ensemble`	OWLv2
`owlv2-base-patch16-finetuned`	`kerasformers/owlv2-base-patch16-finetuned`	OWLv2
`owlv2-large-patch14`	`kerasformers/owlv2-large-patch14`	OWLv2
`owlv2-large-patch14-ensemble`	`kerasformers/owlv2-large-patch14-ensemble`	OWLv2
`owlv2-large-patch14-finetuned`	`kerasformers/owlv2-large-patch14-finetuned`	OWLv2

Tips

Set KERAS_BACKEND before importing Keras / kerasformers.
Prefer Processor.from_weights(...) so image size matches the variant.
Open-vocab thresholds are often much lower than closed-set detectors (try 0.1).
OWLv2 pads to square before resize; pass the original (height, width) as target_sizes carefully (see the OWLv2 docs for the padding trap).
See OWLv2 docs and Loading Weights.
Community / upstream safetensors still work via the hf: prefix, e.g. Owlv2Detect.from_weights("hf:google/owlv2-base-patch16").