OpenGVLab

/

VideoMAEv2-giant

Like 5

Follow

OpenGVLab 2.05k

Hello, and thank you for sharing VideoMAEv2-giant!

With transformers 5 (tested on 5.16.1 with torch 2.5.1), loading the model with trust_remote_code=True fails:

AttributeError: 'VideoMAEv2' object has no attribute 'all_tied_weights_keys'. Did you mean: '_tied_weights_keys'?

transformers 5 expects remote-code models to call self.post_init() at the end of __init__, and the transformers maintainers have said the crash is intentional (huggingface/transformers#43883).

Adding self.post_init() gets the model to load, but extract_features then fails with NotImplementedError: Cannot copy out of meta tensor; no data!. With use_learnable_pos_emb: false, the sinusoidal pos_embed is a plain tensor attribute created in VisionTransformer.__init__ (line 376 of modeling_videomaev2.py). It looks like transformers 5 builds the model on the meta device, so this tensor never gets real data.

These two changes fix it for us:

         else:
             # sine-cosine positional embeddings is on the way
-            self.pos_embed = get_sinusoid_encoding_table(
-                num_patches, embed_dim)
+            with torch.device("cpu"):
+                self.pos_embed = get_sinusoid_encoding_table(
+                    num_patches, embed_dim)

         self.model = VisionTransformer(**self.model_config)
+        self.post_init()

With both changes, the model loads under transformers 5.16.1 and 4.57.6, and extract_features on a fixed random input returns exactly the same output as the unmodified model under 4.57.6 (maximum absolute difference 0). We also tried registering pos_embed as a non-persistent buffer. That loads too, but under transformers 5 the buffer stays uninitialized and the features come out as NaN, so the torch.device("cpu") version seems safer.

The other VideoMAEv2 checkpoints may need the same change, but I have only tested giant.