*This model was published in HF papers on 2026-01-15 and contributed to Hugging Face Transformers on 2026-10-08.*
# Molmo2
[Molmo2](https://huggingface.co/papers/2601.10611) is a family of open-weight vision-language models by AllenAI that are state-of-the-art among open-source models, with exceptional capabilities in point-driven grounding for single image, multi-image, and video tasks. The architecture combines a Vision Transformer (ViT) for image processing with an adapter layer connecting vision and text modalities, and a text decoder based on transformer architecture with rotary position embeddings.
The abstract from the paper is the following:
*Today's strongest video-language models (VLMs) remain proprietary. The strongest open-weight models either rely on synthetic data from proprietary VLMs, effectively distilling from them, or do not disclose their training data or recipe. As a result, the open-source community lacks the foundations needed to improve on the state-of-the-art video (and image) language models. Crucially, many downstream applications require more than just high-level video understanding; they require grounding -- either by pointing or by tracking in pixels. Even proprietary models lack this capability. We present Molmo2, a new family of VLMs that are state-of-the-art among open-source models and demonstrate exceptional new capabilities in point-driven grounding in single image, multi-image, and video tasks. Our key contribution is a collection of 7 new video datasets and 2 multi-image datasets, including a dataset of highly detailed video captions for pre-training, a free-form video Q&A dataset for fine-tuning, a new object tracking dataset with complex queries, and an innovative new video pointing dataset, all collected without the use of closed VLMs. We also present a training recipe for this data utilizing an efficient packing and message-tree encoding scheme, and show bi-directional attention on vision tokens and a novel token-weight strategy improves performance. Our best-in-class 8B model outperforms others in the class of open weight and data models on short videos, counting, and captioning, and is competitive on long-videos. On video-grounding Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on video counting) and surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on video pointing and 56.2 vs 41.1 J&F on video tracking).*
You can find all the original Molmo2 checkpoints under the [Molmo2](https://huggingface.co/collections/allenai/molmo2-67d6b5b0e138c5d621de1e5d) collection.
## Usage example
### Image-text-to-text generation
The example below demonstrates how to generate text based on an image with [`Pipeline`] or the [`AutoModel`] class.
```python
from transformers import pipeline
pipeline = pipeline(
task="image-text-to-text",
model="allenai/Molmo2-8B",
device_map="auto",
)
messages = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
{"type": "text", "text": "Describe this image."},
],
}
]
pipeline(text=messages, max_new_tokens=128, return_full_text=False)
```
```python
from transformers import AutoModelForImageTextToText, AutoProcessor
processor = AutoProcessor.from_pretrained("allenai/Molmo2-8B")
model = AutoModelForImageTextToText.from_pretrained(
"allenai/Molmo2-8B",
device_map="auto",
)
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image."},
{"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
],
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
)
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_text = processor.decode(
generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
)
print(generated_text)
# In this captivating image, a snow leopard is captured ...
```
### Video pointing
Molmo2 can also generate grounded points for video inputs. The points are returned in the generated text as a
`` payload, where each coordinate group contains a timestamp followed by point ids and normalized
image coordinates.
```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "allenai/Molmo2-8B"
video_url = "https://storage.googleapis.com/oe-training-public/demo_videos/many_penguins.mp4"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Point to the penguins."},
{"type": "video", "video": video_url},
],
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
)
generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
input_len = inputs["input_ids"].shape[1]
generated_text = processor.decode(generated_ids[0][input_len:], skip_special_tokens=True)
print(generated_text)
```
## Molmo2Config
[[autodoc]] Molmo2Config
## Molmo2VisionConfig
[[autodoc]] Molmo2VisionConfig
## Molmo2AdapterConfig
[[autodoc]] Molmo2AdapterConfig
## Molmo2TextConfig
[[autodoc]] Molmo2TextConfig
## Molmo2Processor
[[autodoc]] Molmo2Processor
- __call__
## Molmo2ImageProcessor
[[autodoc]] Molmo2ImageProcessor
- __call__
- preprocess
## Molmo2VideoProcessor
[[autodoc]] Molmo2VideoProcessor
- __call__
## Molmo2Model
[[autodoc]] Molmo2Model
- forward
## Molmo2TextModel
[[autodoc]] Molmo2TextModel
- forward
## Molmo2Adapter
[[autodoc]] Molmo2Adapter
- forward
## Molmo2VisionModel
[[autodoc]] Molmo2VisionModel
- forward
## Molmo2ForConditionalGeneration
[[autodoc]] Molmo2ForConditionalGeneration
- forward
