Univariate pre-training addresses different variate numbers of datasets during pre-training. A multivariate extension is crucial for the future improvement of Sundial, including:
Architecture: Recent works proposed new attention mechanisms (e.g., Moirai, Timer-XL) for intra-/inter-variate modeling, which can be seamlessly incorporated into Sundial. Multivariate extension on the flow-matching network (e.g., from MLP to iTransformer) is applicable to make the post-merging on univariate representations.
Post-training: Another roadmap is univariate pre-training and multivariate fine-tuning. Similar to GPT-3, a univariate pre-trained TSFM is a start-point model. We will explore multivariate prompting (e.g., special variate token) to instruct TSFMs on downstream tasks.
How to do that in your code?
How to do that in your code?