Problem about training method

[zcy8998]

Thank you for sharing this excellent work!
I've been experimenting with the approach and observed that while the Transformer architecture supports multi-dataset training, the performance tends to be suboptimal when employing a sequential training strategy across individual datasets.
I would like to kindly ask if there are recommended techniques or best practices to mitigate this issue.

[luckyyf0517]

您好，请问你找到比较好的解决办法了吗？我最近也在做这方面的工作，我的方案是自定义一个sampler来保证每个batch是随机的，并且batch内来自同一个子数据集（避免异构子数据集导致的shape错误）

class PerDatasetBatchSampler(Sampler):
    """
    Sampler that creates batches from a single dataset at a time.

    This sampler:
    1. Supports both distributed and non-distributed training
    2. Shuffles each dataset internally
    3. Creates batches per dataset (keeps incomplete batches by default)
    4. Concatenates batches from all datasets
    5. Distributes batches across multiple GPUs (if distributed)
    6. Ensures different workers/GPUs get different data
    7. For distributed training, drops excess batches to ensure equal distribution

    Args:
        dataset: ConcatCSIDataset
        batch_size: Batch size
        num_replicas: Number of processes (GPUs). Default=1 for non-distributed.
        rank: Rank of current process. Default=0.
        shuffle: Whether to shuffle data
        seed: Random seed
        drop_last_dataset: Whether to drop incomplete batches within each dataset.
            Default=False to keep all data including incomplete final batches.
    """

如果您有更好的方法，可以交流一下😊

[zcy8998]

您好，我们在最近的工作中讨论了四种不同的针对异构CSI的训练方案，代码已经开源在我的github主页，如果感兴趣可以看下～