Add V-JEPA 2.1 inference support by davevanveen · Pull Request #45497 · huggingface/transformers

davevanveen · 2026-04-17T22:05:24Z

What does this PR do?

Adds inference support and checkpoint conversion for Meta V-JEPA 2.1 pretrained backbones within the existing vjepa2 model family.

V-JEPA 2.1 was released by Meta on 2026-03-16 with four pretrained encoders at 384 resolution: ViT-B (80M), ViT-L (300M), ViT-g (1B), and ViT-G (2B). This PR extends the existing VJEPA2Model with backward-compatible config fields and modeling changes to support loading these checkpoints.

Changes

Config (configuration_vjepa2.py): 8 new fields with backward-compatible defaults that preserve V-JEPA 2.0 behavior:

use_rope_interleave, use_modality_embeddings, interpolate_rope, return_all_tokens, img_temporal_dim_size
Predictor-only: teacher_embed_dim, n_output_distillation, hierarchical_layers

Modeling (modeling_vjepa2.py):

Corrected RoPE implementation (repeat_interleave vs repeat) with config toggle
Learnable modality embeddings (image/video) for both encoder and predictor, with top-level modality routing
Separate image patch embedding (tubelet_size=1) when img_temporal_dim_size is set
Hierarchical feature extraction with per-layer norms_block in encoder
Smart encoder output: concatenated features when predictor needs them (n_output_distillation > 1), single-norm output for get_vision_features() / skip_predictor=True
RoPE position interpolation for flexible resolution (encoder only, matches Meta's behavior)
Multi-layer predictor embed (Linear+GELU+Linear) for n_output_distillation > 1
Predictor context token projection (proj_context) for return_all_tokens
Teacher embedding dimension-based output projection sizing
VJEPA2ForVideoClassification guard against unsupported hierarchical configs

Converter (convert_vjepa2_to_hf.py):

Four new 2.1 model variants with correct architecture parameters
Key remappings for all new layers (modality embeddings, norms_block, patch_embed_img, proj_context)
Checkpoint key handling (ema_encoder for distilled models, target_encoder for self-supervised)
Updated test function to handle 2.1 tuple predictor returns

Tests (test_modeling_vjepa2.py):

Config defaults test (verifies 2.0 backward compatibility)
Fast forward pass with shape assertions for n_output_distillation=1 (distilled path)
Fast forward pass with shape assertions for n_output_distillation=4 (multi-layer path)
All 90 existing tests pass unchanged

Docs (vjepa2.md): V-JEPA 2.1 section with checkpoint table and architecture notes.

Verification

Verified end-to-end against Meta's reference implementation on two checkpoints covering both architecture paths:

ViT-B/384 (80M, n_output_distillation=1, distilled):

All weight keys match (strict load)
Encoder: max diff 0.0001
Predictor: max diff 0.008

ViT-g/384 (1B, n_output_distillation=4, self-supervised):

All weight keys match (strict load)
Encoder: max diff 0.004
Predictor target: max diff 0.0002
Predictor context: max diff 0.002

All diffs are within SDPA floating-point precision.

2.1 checkpoint summary

Model	Params	Distilled	`n_output_distillation`	`teacher_embed_dim`
ViT-B/16, 384	80M	Yes (ViT-G)	1	1664
ViT-L/16, 384	300M	Yes (ViT-G)	1	1664
ViT-g/16, 384	1B	No	4	—
ViT-G/16, 384	2B	No	4	—

Fixes #45496

Code Agent Policy

I confirm that this is not a pure code agent PR.

AI assistance (Claude Code) was used for implementation. I reviewed and verified all changes, ran tests, and validated checkpoint conversion end-to-end on both architecture paths (ViT-B and ViT-g).

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Add V-JEPA 2.1 inference support #45496
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

Who can review?

@yonigozlan — vision models reviewer and original V-JEPA 2 contributor
@molbap — vision models reviewer

extend VJEPA2Config with backward-compatible fields for 2.1: use_rope_interleave, use_modality_embeddings, interpolate_rope, return_all_tokens, img_temporal_dim_size, teacher_embed_dim, n_output_distillation, hierarchical_layers. modeling changes: - rope interleave toggle (repeat vs repeat_interleave) - rope position interpolation for flexible resolution - modality embeddings (img_mod_embed/video_mod_embed) - separate image patch embedding (tubelet_size=1) - hierarchical feature extraction with per-layer norms_block - multi-layer predictor embed (mlp when n_output_distillation > 1) - predictor proj_context for return_all_tokens - teacher_embed_dim-based output projection sizing

add four 2.1 variants (vit_base, vit_large, vit_giant, vit_gigantic) with correct configs including hierarchical_layers, teacher_embed_dim, n_output_distillation, and all 2.1-specific flags. handle 2.1 key remappings: patch_embed_img, norms_block, modality embeddings, predictor_proj_context, and ema_encoder checkpoint key.

- config defaults test ensuring 2.0 backward compatibility - fast forward pass test with tiny 2.1 config (n_output_distillation=1) - fast forward pass test with multi-layer distillation (n_output_distillation=4) - slow integration smoke test for 2.1-like config

document four 2.1 pretrained checkpoints, key architectural differences from 2.0, and note distilled variants.

- fix rope interpolation: use pretrained_grid_size (256/patch_size) as numerator, not current grid_size. disable for predictor attention. - fix predictor proj output dim: use n_hier (all hierarchical layers) not n_dist for projection size - add predictor modality embeddings (separate from encoder's) - add predictor modality embedding key mapping in converter - pass is_predictor flag through attention to disable rope interpolation verified against meta reference: encoder max diff 0.0001, predictor max diff 0.008 (sdpa vs eager precision).

raise ValueError when n_output_distillation > 1 since the pooler and classifier expect hidden_size but hierarchical models produce hidden_size * n_dist. no 2.1 classification checkpoints exist yet.

for hierarchical models (n_output_distillation > 1), the encoder now returns norms_block[-1](x) when skip_predictor=True, matching meta's default inference behavior. concatenated features are only returned when the predictor needs them.

…tion determine is_image at VJEPA2Model.forward level by comparing pixel_values_videos.shape[1] to config.img_temporal_dim_size. pass explicit boolean to predictor to select img_mod_embed vs video_mod_embed. previously always used video_mod_embed.

@strict

replace config.to_dict() spread with copy.copy(config) to avoid potential issues with @strict decorator rejecting metadata keys.

remove unused VJEPA2_1_CHECKPOINT_KEYS dict, add comment noting provisional hub model names, add pretrained_grid_size comment.

for 2.1 models with return_all_tokens=True, meta's predictor returns (target, context) tuple while hf returns [context, target] concatenated. compare slices separately with atol=1e-2 for sdpa precision. set encoder.return_hierarchical=True for n_dist>1 models.

verify encoder and predictor output shapes for both n_dist=1 (single-norm, teacher_embed_dim projection) and n_dist=4 (concatenated hierarchical, hidden_size projection) paths.

yonigozlan · 2026-05-01T14:45:59Z

Hello @davevanveen ! Thanks a lot for opening this PR. Sorry for the delay in reviewing this, we have been very busy with lots of new models to review and recent core changes. I'll try to get to this PR next week!

davevanveen · 2026-05-01T23:05:43Z

Appreciate the note, @yonigozlan! No rush from me; I imagine y'all are pretty slammed. Thanks for all the work you do. Cheers.

yonigozlan

Hey @davevanveen ! Thanks a lot for working on this :), I have a few comments, mainly on possible simplifications, and I also think we need a better API to make sure the outputs are coherent between VJEPA2 and 2.1

yonigozlan · 2026-05-08T20:54:03Z

+            img_config = copy.copy(config)
+            img_config.tubelet_size = 1
+            self.patch_embeddings_img = VJEPA2PatchEmbeddings3D(img_config, hidden_size=hidden_size)


Let's add a tubelet_size arg to VJEPA2PatchEmbeddings3D instead of this workaround, and take the config value when the arg is not set.

yonigozlan · 2026-05-08T20:56:44Z

-        # then embedding lookup fails. In these cases, we duplicate the frames.
-        if num_frames < self.config.tubelet_size:
-            pixel_values_videos = pixel_values_videos.repeat(1, 1, self.config.tubelet_size, 1, 1)
+        is_image = self.config.img_temporal_dim_size is not None and num_frames == self.config.img_temporal_dim_size


Is config.img_temporal_dim_size ever something other than 1? otherwise let's change img_temporal_dim_size to a boolean flag in the config like use_image_patch_embedder and simplify this logic

Confirmed it's always 1. Renamed to use_image_patch_embedder: bool (default False) and simplified to is_image = self.config.use_image_patch_embedder and num_frames == 1.

yonigozlan · 2026-05-08T21:02:06Z

        config: VJEPA2Config,
        hidden_size: int = 1024,
        num_attention_heads: int = 16,
+        is_predictor: bool = False,


Looks like is_predictor is only used to interpolate rope or not, so let's use an interpolate_rope flag instead (which would override the config one if specified)

yonigozlan · 2026-05-08T21:04:44Z

-        # Remove frame component from ids (1st term) and height component (2nd term)
        width_ids = (ids - tokens_per_frame * frame_ids) - tokens_per_row * height_ids
+
+        if self.config.interpolate_rope and not self.is_predictor and self.grid_size > 1:


We would only need this then:

Suggested change

if self.config.interpolate_rope and not self.is_predictor and self.grid_size > 1:

if self.interpolate_rope and self.grid_size > 1:

yonigozlan · 2026-05-08T21:11:07Z

+        When set, creates a separate image patch embedding with `tubelet_size=1`.
+    teacher_embed_dim (`int`, *optional*, defaults to `None`):
+        Teacher embedding dimension for distilled models. Controls predictor output projection size.
+    n_output_distillation (`int`, *optional*, defaults to 0):


Let's call this num_distillation_outputs, and set it to 1 by default, so that we don't need the "> 0" check in the modeling code (no need to have a redundant condition when hierarchical_layers is set)

Done. Renamed to num_distillation_outputs with default 1. Also took the opportunity to rename return_all_tokens to use_context_projection, since after the predictor refactor the old name described deleted behavior (the concatenation into last_hidden_state). Happy to revert if you'd rather keep return_all_tokens.

yonigozlan · 2026-05-09T18:29:40Z

+            N_ctxt = context_mask[0].shape[1]
+            hf_context = predictor_outputs.last_hidden_state[:, :N_ctxt]
+            hf_target = predictor_outputs.last_hidden_state[:, N_ctxt:]
+            assert torch.allclose(hf_target, og_target, atol=1e-2)
+            assert torch.allclose(hf_context, og_context, atol=1e-2)


This is not the ideal API imo, We could instead change VJEPA2WithMaskedInputPredictorOutput in the modeling code to add a context_predictions field so that, we'll have:

Suggested change

N_ctxt = context_mask[0].shape[1]

hf_context = predictor_outputs.last_hidden_state[:, :N_ctxt]

hf_target = predictor_outputs.last_hidden_state[:, N_ctxt:]

assert torch.allclose(hf_target, og_target, atol=1e-2)

assert torch.allclose(hf_context, og_context, atol=1e-2)

assert torch.allclose(predictor_outputs.last_hidden_state, og_target, atol=1e-2)

assert torch.allclose(predictor_outputs.context_predictions, og_context, atol=1e-2)

Done at both sites. Validated end-to-end against Meta's vit_base_2_1_384 and vit_giant_2_1_384 checkpoints. allclose passes for full-mask and partial-mask cases.

yonigozlan · 2026-05-11T14:18:55Z

+            N_ctxt = context_mask[0].shape[1]
+            hf_context = predictor_outputs.last_hidden_state[:, :N_ctxt]
+            hf_target = predictor_outputs.last_hidden_state[:, N_ctxt:]
+            assert torch.allclose(hf_target, og_target, atol=1e-2)
+            assert torch.allclose(hf_context, og_context, atol=1e-2)


Same here:

Suggested change

N_ctxt = context_mask[0].shape[1]

hf_context = predictor_outputs.last_hidden_state[:, :N_ctxt]

hf_target = predictor_outputs.last_hidden_state[:, N_ctxt:]

assert torch.allclose(hf_target, og_target, atol=1e-2)

assert torch.allclose(hf_context, og_context, atol=1e-2)

assert torch.allclose(predictor_outputs.last_hidden_state, og_target, atol=1e-2)

assert torch.allclose(predictor_outputs.context_predictions, og_context, atol=1e-2)

yonigozlan · 2026-05-11T14:24:48Z

        else:
            predictor_output = None

        encoder_output = VJEPA2WithMaskedInputModelOutput(


We'll also need to add a hierarchical_hidden_state field here as well since it wouldn't be in last_hidden_state anymore

Done. Added hierarchical_hidden_state to VJEPA2WithMaskedInputPredictorOutput. Also added it to VJEPA2WithMaskedInputModelOutput so users calling VJEPA2Model(..., skip_predictor=True) for feature extraction can still reach hierarchical features when predictor_output is None.

yonigozlan · 2026-05-11T14:26:05Z

            )
            predictor_output = VJEPA2WithMaskedInputPredictorOutput(
                last_hidden_state=predictor_outputs.last_hidden_state,
                target_hidden_state=apply_masks(sequence_output, target_mask),


Same here (hierarchical_hidden_state if not None, otherwise sequence_output)

yonigozlan · 2026-05-11T14:26:52Z

+        if config.n_output_distillation > 1:
+            raise ValueError(
+                f"Classification heads for hierarchical distillation outputs "
+                f"(n_output_distillation={config.n_output_distillation}) are not yet supported. "
+                f"Use VJEPA2Model for feature extraction instead."
+            )


Why have this guard here? I think we can get rid of it fully and let users experiment

Guard removed.

Added a small smoke test (test_classification_head_with_hierarchical_distillation) that instantiates VJEPA2ForVideoClassification with a 2.1-style config (num_distillation_outputs=4, hierarchical_layers set) and runs a forward pass. The pooler consumes last_hidden_state (post-loop, dim hidden_size), not hierarchical_hidden_state, so shapes work out for any num_distillation_outputs.

Documented in the classification head's docstring that for V-JEPA 2.1 the pooler sees unnormalized features. Users wanting hierarchical features should use VJEPA2Model for feature extraction.

… guard

davevanveen · 2026-06-03T22:20:54Z

Hi @yonigozlan - Thanks again for the detailed review! I addressed all requested changes three weeks ago. When you have a chance, I'd appreciate another look. Happy to address any loose ends. Cheers.

molbap · 2026-06-04T12:40:44Z

Hey @davevanveen ! Me and/or @guarin will follow-up on reviewing the PR soon, sorry for the delay 🤗

molbap

Thanks for iterating on this! I added a couple questions + discussion about checkpoint upload. Let me know!

molbap · 2026-06-09T17:12:42Z

+    @slow
+    def test_inference_vjepa2_1_base(self):
+        """Smoke test: instantiate a 2.1-like config and run forward pass."""
+        config = VJEPA2Config(


Since it's small it doesn't need @slow (it just asserts shapes so it does not bring much in current state) but let's add an integration test!

The integration test should be comparing against real outputs taken from the original implementation by Meta.

That way you can push the converted checkpoints to an organisation you own on the hub (still indicating Meta's license) and we can see to move it to their org later on (your authorship/initial upload is preserved). What do you think?

Removed @slow from test_inference_vjepa2_1_base.

Great idea on the integration test! Curious what you think of this approach.

Added test_inference_vjepa2_1_integration

Loads davevanveen/vjepa2.1-vitb-fpc64-384

Runs a forward pass on the standard COCO fixture image

Asserts hierarchical_hidden_state[:, :4, :4] and predictor_output.last_hidden_state[:, :4, :4] against values from Meta's original implementation.

Converted vit_base_2_1_384 and pushed to davevanveen/vjepa2.1-vitb-fpc64-384 (link). All allclose assertions passed.

Note: context_predictions required atol=2e-2 rather than 1e-2. Max diff ~0.014 and mean ~2e-6 suggest a float32 accumulation noise across the predictor layers, not a weight error. I updated converter accordingly.

Happy to transfer to Meta's org as you see fit.

molbap · 2026-06-10T09:23:40Z

output[-1] will not be VJEPA2WithMaskedInputPredictorOutput but a FloatTensor here

Fixed. Replaced with the canonical recursive pattern used by CLIP, BLIP-2, etc.:

def to_tuple(self) -> tuple[Any]: return tuple(v.to_tuple() if isinstance(v, ModelOutput) else v for v in self.values())

molbap · 2026-06-10T09:25:45Z

        argsort = torch.argsort(position_masks, dim=1)  # [B, N]
        hidden_states, position_masks = self.sort_tokens(hidden_states, position_masks, argsort)

+        if self.config.use_modality_embeddings and hasattr(self.embeddings, "video_mod_embed"):


if we use_modality_embeddings we always have video_mod_embed has attribute no?

Correct. VJEPA2PredictorEmbeddings.__init__ always creates both img_mod_embed and video_mod_embed when use_modality_embeddings=True. Removed the redundant hasattr guard.

molbap · 2026-06-10T09:35:35Z

+def _is_2_1_model(model_name):
+    return "2_1" in model_name


does not warrant a function

Removed the helper entirely. All three call sites now inline "2_1" in model_name directly.

molbap · 2026-06-10T10:03:39Z

On that, see my comment on tests about uploading the converted checkpoints to the hub!

See above comment.

molbap · 2026-06-10T10:05:30Z

+        target_predictions = self.proj(hidden_states[:, N_ctxt:])
+        context_predictions = self.proj_context(hidden_states[:, :N_ctxt]) if self.proj_context is not None else None


let's use the occasion to get rid of namings like N_ctxt (we can afford full/explicit wording)

Renamed to num_context_tokens throughout VJEPA2Predictor.forward. Also replaced the now-unused D dimension with _.

molbap · 2026-06-10T10:05:58Z

        if context_mask is None and target_mask is None:
            B = pixel_values_videos.size(0)
-            N = sequence_output.size(1)  # ensure we are using dynamic patch size
+            N = sequence_output.size(1)


why delete this, not true anymore?

Restored. You're right, better to clarify: N is defined at runtime to support variable-resolution inputs.

…tore dynamic-N comment

…kpoint

davevanveen · 2026-06-11T21:38:24Z

Hi @molbap, appreciate your review! Great suggestions; all are addressed above. Happy to continue the discussion if anything looks off.

molbap

Sorry for the delay, summer is busy. Thanks for iterating!

molbap · 2026-07-06T10:25:26Z

+            self.img_mod_embed = nn.Parameter(torch.zeros(1, 1, hidden_size))
+            self.video_mod_embed = nn.Parameter(torch.zeros(1, 1, hidden_size))


these are not present in init_weights, and should be I think since they are additive

Done. Added normal_(std=1e-6) initialization for both img_mod_embed and video_mod_embed in _init_weights, covering both VJEPA2Embeddings (encoder) and VJEPA2PredictorEmbeddings (predictor). Used std=1e-6 to match Meta's original implementation, which allocates the parameter as zeros and immediately applies nn.init.normal_(..., std=1e-6).

molbap · 2026-07-06T10:26:46Z

+            hierarchical_hidden_state=hierarchical_hidden_state,
        )

        return encoder_output

    def get_vision_features(self, pixel_values_videos) -> torch.Tensor:
        encoder_output = self.forward(pixel_values_videos, skip_predictor=True)
        return encoder_output.last_hidden_state


we should return hierarchical_hidden_state when it exists, right?

Done. get_vision_features now returns hierarchical_hidden_state when non-None, falling back to last_hidden_state otherwise.

molbap · 2026-07-06T10:27:54Z

+
+        # for vjepa2.1 models, the encoder output to compare is hierarchical_hidden_state
+        # (mirrors the converter's validation: hf_encoder = outputs.hierarchical_hidden_state ?? outputs.last_hidden_state)
+        # reference values from Meta's original implementation — see workstream_001_build.md


this file isn't in the repo (and should not be), let's use Expectations instead of raw values here

Done. Wrapped both expected slices in Expectations({(None, None): [...]}) and switched to torch.testing.assert_close. Removed the stale comment.

molbap · 2026-07-06T10:34:07Z

-        target_dtype = self.patch_embeddings.proj.weight.dtype
-        pixel_values_videos = pixel_values_videos.to(dtype=target_dtype)
-        embeddings = self.patch_embeddings(pixel_values_videos)
+        if is_image and self.patch_embeddings_img is not None:


is_image already requires config.use_image_patch_embedder no?

Correct. Removed the redundant guard. is_image already requires use_image_patch_embedder=True, which is the same condition under which patch_embeddings_img is created, so the is not None check is unreachable.

molbap · 2026-07-06T10:34:54Z

+        # matches Meta's hardcoded RoPE reference resolution (256 for patch_size=16)
+        self.pretrained_grid_size = 256 // self.config.patch_size


this should go into the configuration, in the recent rope_parameters

Done. Added pretrained_grid_size: int | None = None to VJEPA2Config alongside the existing RoPE fields (use_rope_interleave, interpolate_rope). Falls back to 256 // patch_size when unset, preserving existing behavior.

An alternative to the flat-field approach above would be adopting the rope_parameters: dict | None = None dict pattern used by other models and place pretrained_grid_size inside it. VJEPA2's custom RoPE doesn't use the shared ROPE_INIT_FUNCTIONS pipeline, so the dict would be purely a namespace here. Happy to switch if you prefer.

molbap · 2026-07-06T10:35:20Z

        width_ids = (ids - tokens_per_frame * frame_ids) - tokens_per_row * height_ids
+
+        if self.interpolate_rope and self.grid_size > 1:
+            h_scale = (self.pretrained_grid_size - 1.0) / max(self.grid_size - 1.0, 1.0)


same value for h_ and w_ scale, one suffice

Done. Collapsed to a single scale variable.

molbap · 2026-07-06T10:36:02Z

+    @slow
+    def test_inference_vjepa2_1_integration(self):
+        """Integration test: load vjepa2.1-vitb from Hub and compare encoder output against Meta's original implementation."""
+        model = VJEPA2Model.from_pretrained("davevanveen/vjepa2.1-vitb-fpc64-384").to(torch_device)


let's settle that before merge :) I think we can open a hub PR

Happy to open a Hub PR from davevanveen/vjepa2.1-vitb-fpc64-384 into the facebook org. The converted checkpoint is ready. The prerequisite is that the destination repo exists first.

@molbap and/or @koustuvsinha - would you be able to create a repo for the V-JEPA 2.1 checkpoints, e.g. facebook/vjepa2.1-vitb-fpc64-384? Once that's done, I will open the Hub PR.

Note that community ports are already appearing (e.g. the collection referenced in this issue). An official checkpoint under the facebook org would give users a clear canonical source.

molbap · 2026-07-06T10:37:14Z

        expected_shape = torch.Size((1, num_masks, 1024))
        self.assertEqual(outputs.predictor_output.last_hidden_state.shape, expected_shape)

+    def test_inference_vjepa2_1_base(self):


this is same as test_model_2_1_forward and is a weak test (only test not-None), I think we can delete

Done. Deleted test_inference_vjepa2_1_base.

molbap · 2026-07-06T10:38:08Z

+    use_context_projection: bool = False
+    use_image_patch_embedder: bool = False
+    teacher_embed_dim: int | None = None
+    num_distillation_outputs: int = 1


for the issue where num_distillation_outputs > len(hierarchical_layers) you can add a strict validation in the config

Done. Added a __post_init__ validation that raises ValueError when num_distillation_outputs > len(hierarchical_layers). No-op when hierarchical_layers is None (V-JEPA 2.0 configs).

… guard, collapse scale

…ion_outputs validation

…lback and config validation

davevanveen · 2026-08-24T16:26:53Z

Hi @molbap @yonigozlan - Everything is passing and ready on my end. Let me know if you’d like any further changes before we merge. Thanks.

github-actions · 2026-08-24T16:28:42Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: vjepa2

github-actions · 2026-08-24T16:34:26Z

CI recap

Dashboard: View test results in Grafana
Latest run: 30168715675:2
Result: success | Jobs: 4 | Tests: 283 | Failures: 0 | Duration: 2s

davevanveen added 13 commits April 16, 2026 21:33

docs: add v-jepa 2.1 section to model documentation

ecb804d

document four 2.1 pretrained checkpoints, key architectural differences from 2.0, and note distilled variants.

style: apply ruff formatting to modeling_vjepa2.py

441a37b

fix(vjepa2): guard classification head against hierarchical configs

b9480e6

raise ValueError when n_output_distillation > 1 since the pooler and classifier expect hidden_size but hierarchical models produce hidden_size * n_dist. no 2.1 classification checkpoints exist yet.

fix(vjepa2): use shallow copy for image patch embed config

f52c42e

replace config.to_dict() spread with copy.copy(config) to avoid potential issues with @strict decorator rejecting metadata keys.

refactor(vjepa2): remove dead code and add inline comments

61dcb68

remove unused VJEPA2_1_CHECKPOINT_KEYS dict, add comment noting provisional hub model names, add pretrained_grid_size comment.

test(vjepa2): add output shape assertions for 2.1 models

140e7c0

verify encoder and predictor output shapes for both n_dist=1 (single-norm, teacher_embed_dim projection) and n_dist=4 (concatenated hierarchical, hidden_size projection) paths.

davevanveen mentioned this pull request Apr 17, 2026

Release V-JEPA 2.1 on Hugging Face facebookresearch/vjepa2#137

Open

Merge branch 'main' into add-vjepa2.1-support

9b7468c

evalstate mentioned this pull request Apr 28, 2026

Cumulative defect fixes from recent Transformers PRs evalstate/transformers#41

Open

Samoed mentioned this pull request Apr 29, 2026

Add model: VJepa2.1 embeddings-benchmark/mteb#4559

Open

This was referenced Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open

Cumulative defect fixes from recent Transformers PRs evalstate/transformers#43

Open

Merge branch 'main' into add-vjepa2.1-support

6753951

Merge branch 'main' into add-vjepa2.1-support

af34f63

yonigozlan reviewed May 11, 2026

View reviewed changes

davevanveen added 6 commits May 15, 2026 21:55

refactor: rename vjepa2 config fields for 2.1 api clarity

c19b40a

merge origin/add-vjepa2.1-support into local refactor

2b38426

refactor: simplify vjepa2 patch embedding and rope flags

a070033

feat: add explicit vjepa2 encoder and predictor outputs

4f4969f

refactor: simplify vjepa2 distillation handling and remove classifier…

8f10aeb

… guard

chore: update vjepa2.1 conversion checks and tests

41e1065

davevanveen added 2 commits May 31, 2026 14:37

Merge branch 'main' into add-vjepa2.1-support

2e7e3e7

Merge branch 'main' into add-vjepa2.1-support

4fa8096

molbap self-requested a review June 4, 2026 12:47

molbap reviewed Jun 10, 2026

View reviewed changes

davevanveen added 7 commits June 11, 2026 17:28

fix(vjepa2): inline _is_2_1_model helper

771b563

fix(vjepa2): fix to_tuple, rename N_ctxt, drop redundant hasattr, res…

5985574

…tore dynamic-N comment

test(vjepa2): remove @slow from synthetic vjepa2.1 config test

d5ddede

test(vjepa2): add integration test for vjepa2.1-vitb against hub chec…

b7470c0

…kpoint

test(vjepa2): assert predictor output in vjepa2.1 integration test

819e4bf

fix(vjepa2): relax context_predictions converter tolerance to 2e-2

955debc

Merge branch 'main' into add-vjepa2.1-support

c1d6f76

davevanveen added 2 commits June 18, 2026 16:29

Merge branch 'main' into add-vjepa2.1-support

c9266d1

Merge branch 'main' into add-vjepa2.1-support

0965ec3

molbap reviewed Jul 6, 2026

View reviewed changes

davevanveen added 7 commits July 22, 2026 09:30

Merge branch 'main' into add-vjepa2.1-support

4d90ecb

test(vjepa2): use Expectations, remove weak test

e8c0b11

fix(vjepa2): init mod embeds, fix get_vision_features, drop redundant…

2fcdab0

… guard, collapse scale

fix(vjepa2): move pretrained_grid_size into config, add num_distillat…

daf16fa

…ion_outputs validation

test(vjepa2): add unit tests for get_vision_features hierarchical fal…

ebc2798

…lback and config validation

Merge branch 'main' into add-vjepa2.1-support

cf00fd2

Merge branch 'main' into add-vjepa2.1-support

ae8a3b9

Merge branch 'main' into add-vjepa2.1-support

974de69

ArthurZucker added the New model label Sep 1, 2026

ArthurZucker requested a review from molbap September 1, 2026 06:21

	if self.config.interpolate_rope and not self.is_predictor and self.grid_size > 1:
	if self.interpolate_rope and self.grid_size > 1:

		target_predictions = self.proj(hidden_states[:, N_ctxt:])
		context_predictions = self.proj_context(hidden_states[:, :N_ctxt]) if self.proj_context is not None else None

		self.img_mod_embed = nn.Parameter(torch.zeros(1, 1, hidden_size))
		self.video_mod_embed = nn.Parameter(torch.zeros(1, 1, hidden_size))

		# matches Meta's hardcoded RoPE reference resolution (256 for patch_size=16)
		self.pretrained_grid_size = 256 // self.config.patch_size

Conversation

davevanveen commented Apr 17, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Changes

Verification

2.1 checkpoint summary

Code Agent Policy

Before submitting

Who can review?

Uh oh!

yonigozlan commented May 1, 2026

Uh oh!

davevanveen commented May 1, 2026

Uh oh!

yonigozlan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davevanveen commented Jun 3, 2026

Uh oh!

molbap commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davevanveen commented Apr 17, 2026 •

edited by github-actions Bot

Loading

molbap commented Jun 4, 2026 •

edited

Loading

davevanveen Jun 11, 2026 •

edited

Loading