SONAR training dataset in Huggingface

[jasonrichdarmawan]

Hi,

My goal is to train Sparse Autoencoders (SAEs). If the team can share the training dataset in huggingface, then it would be possible to train SAEs that learn features from the training dataset

The problem:

1. To train SAEs, you need to use the dataset that was used to train SONAR (i.e., SAEs requires the same data distribution), or the dataset that was used to train NLLB. However, the allenai/nllb dataset available in Huggingface is noisy

[avidale]

Hi @jasonrichdarmawan!
SONAR has been trained on the NLLB training data (see https://github.com/facebookresearch/fairseq/tree/nllb?tab=readme-ov-file#datasets)
It is composed of three main sources:

• Primary data: most of it is described here
• Mined data: this is the HF dataset that you refer to, but it needs filtering, see the NLLB paper and the Stopes repo for the description and implementation of the concrete filters
• Backtranslated data: see the instructions on generating it here.

Overall, I would say that reconstructing the full NLLB training dataset is a laborious and expensive process, but, unfortunately, we cannot release this data directly.

[jasonrichdarmawan]

@avidale

Would you mind to confirm whether the team preprocess the dataset or does the team use identical NLLB dataset for training? Ah you have answered it. Sorry, I was reading it on my phone and didn't read the full comments

For context, I assume allenai/nllb/Eng-Latn_zsm-Latn dataset was constructed from the link you provided[1]

[1] initial data analysis show that the dataset is noisy (the translation target's meaning is not correct; surprisingly the prediction is somewhat more correct) as shown in the screenshot above

I intend to use nllb/allenai dataset to train the SAEs instead of constructing from the provided scripts[2] because I want to leverage the dataset.load_dataset(streaming=True) to save the disk space and also to only take few samples from each language for training

[2] one of the script comment claimed 45GB, that's a lot. I am not sure how long it would take to train with 1x A4000. That's why the streaming mode is a requirement for me

[jasonrichdarmawan]

Ah you have answered it. Sorry, I was reading it on my phone and didn't read the full comments

[avidale]

I want to leverage the dataset.load_dataset(streaming=True) to save the disk space and also to only take few samples from each language for training

A while ago, I randomly selected 10M samples from the NLLB mined dataset and annotated them with the BLASER-2 quality estimation model: https://huggingface.co/datasets/slone/nllb-200-10M-sample. If you take this dataset and load it in the streaming mode with a filter like blaser_sim>=3.5, you will get a cleaner representation of the NLLB mined data.

But please note that you will never reproduce the NLLB model training with the mined dataset only; you need to re-compile the primary data as well to achieve the translation quality that gets close to that.

[jasonrichdarmawan]

Thank you!

Part of the dataset would do just fine. I will use your annotated dataset

As long as the data distribution is the same i.e. the dataset was used for training the SONAR model

I will read the NLLB paper tomorrow morning. I hope:

1. the term "Primary data" does not mean the dataset dominated the training
2. it is possible to create part of the "Primary data" without downloading the entire dataset

[avidale]

In the NLLB terminology, “primary data” is a collection of numerous publicly available (third party) parallel datasets, which were used for training intermediate models for mining and backtranslation (hence the term “primary”). The link above offers download scripts for abour 22 of them, and many more are downloadable through the OPUS website (or, equivalently, through the mtdata command line tool). You can pick any of them, depending on your languages of interest and data appetite.

I am not sure that the exhaustive list of all datasets used for NLLB training is published anywhere, though.

[jasonrichdarmawan]

Hi,

Would you mind to have discussion on how to train SAE?

For context: I am thinking to stop here because the latent's explanations from the trained SAE are not useful to understand how SONAR constructed the embedding and I need to have tangible results

=== ===

What I've tried:

1. Trained SAE with 10M randomly sampled from NLLB dataset. The dataset is filtered by blaser_sim >= 3.5. The filtered dataset. Unfortunately, the SAE latent dimensions are active on sentence with domain-specific content instead of abstract/fundamental linguistic features
2. Trained SAE with part of NLLB primary datasets (except the ones with 20 GB / 11 million samples dataset). The processed dataset

But the result is the same:

1. nllb-200-6m-sample-embedding: SAE latent dimensions are active on sentence with domain-specific content, specifically biblical content
2. nllb-200-6m-sample-embedding + nllb-primary-datasets-embedding: SAE latent dimensions are active on sentence with domain-specific content, with less emphasize on biblical content, though the latent that active on biblical content also active on unrelated sentence

[avidale]

Why do you think this result is bad?
To me, SAE focusing on domains makes sense: to represent an arbitrary sentence, it feels reasonable to start with the genre of the text (whether it is e.g. informal speech, a scientific text, a poem, etc.) and its topic, and focus on the specific content of the sentence only after that.
I am not an expert on SAE, though.

[jasonrichdarmawan]

Why do you think this result is bad?
The goal is to understand how SONAR text2vec model construct its embeddings. If it fires on unrelated domain-specific content, I assume the trained SAEs fails to disentangle the features

Also, currently a word "cat" will active at least 30 SAE latents. I assume that's a lot

To me, SAE focusing on domains makes sense: to represent an arbitrary sentence, it feels reasonable to start with the genre of the text (whether it is e.g. informal speech, a scientific text, a poem, etc.) and its topic, and focus on the specific content of the sentence only after that.

I agree

I am not an expert on SAE, though.

Same