how can train vision encoder with siglip2?

[ctgushiwei]

How to use Siglip to train a visual encoder"

[lucasjinreal]

Hi, this is exactly our v2 doing. Stay tuned for our v2 model release, the training codes will also be available when v2 training finished.

[machinelearningintamil]

@lucasjinreal How do I train the model now? Could you share a sample dataset format? Do I need to follow the sample format to train the model?

This is huggingface sample dataset.

{'chosen': [{'content': [{'text': "In the image, there's a dynamic scene at what appears to be a beach or surfing location. The main focus is on a person skillfully riding a wave on a surfboard. This individual is dressed in a yellow shirt and seems to be enjoying the activity. In addition to the surfer, there are other elements in the scene such as waves breaking and creating white foam, indicating the active nature of the water. Nearby, there's another surfboard floating on the surface of the water, suggesting that more people might be participating in surfing or waiting for their turn. The overall atmosphere conveys a sense of fun, adventure, and connection with nature.",
     'type': 'text'}],
   'role': 'assistant'}],
 'rejected': [{'content': [{'text': 'In the image, there is a man enthusiastically surfing on a wave in the ocean. He is wearing a yellow shirt and short pants, adding a vibrant color contrast to the scene. The surfer is skillfully balancing himself on his surfboard as he rides the wave. Additionally, there are other surfers and surfboards scattered throughout the water, indicating that this might be a popular spot for surfing. The overall atmosphere suggests an active and fun beach day with people enjoying the watersport.',
     'type': 'text'}],
   'role': 'assistant'}],
 'images': [<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x428>],
 'prompt': [{'content': [{'text': None, 'type': 'image'},
    {'text': 'Provide an intricate description of every entity in the image.',
     'type': 'text'}],
   'role': 'user'}]}

[lucasjinreal]

Hi, for sft, just using llava format is OK. for R1 training, we are providing an example these days.

[ctgushiwei]

Hi，When will the code for training Siglip2 be released?

[lucasjinreal]

Yes, will release to support Siglip2 these days. Stay tuned! Our Siglip2 trained model already posted on hf, but supported modeling will be released later on.

[ctgushiwei]

Hi，When will the code for training Siglip2 be released these days.?

[lucasjinreal]

Hi, actually we had trained siglip2 for several versions, will opensource siglip2 training code this week