From 25cacb99135d87001eb1f2761c5e41da7df3a17a Mon Sep 17 00:00:00 2001 From: Pius Fung Date: Mon, 11 Aug 2025 15:06:38 -0700 Subject: [PATCH 1/3] Added information about the use of slow tokenizers Added information about the use of slow tokenizers to generate vocab files in ML. --- explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md b/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md index 485b9505d3..297b29cd68 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md @@ -15,7 +15,7 @@ products: The minimum dedicated ML node size for deploying and using the {{nlp}} models is 16 GB in {{ech}} if [deployment autoscaling](../../../deploy-manage/autoscaling.md) is turned off. Turning on autoscaling is recommended because it allows your deployment to dynamically adjust resources based on demand. Better performance can be achieved by using more allocations or more threads per allocation, which requires bigger ML nodes. Autoscaling provides bigger nodes when required. If autoscaling is turned off, you must provide suitably sized nodes yourself. :::: -The {{stack-ml-features}} support transformer models that conform to the standard BERT model interface and use the WordPiece tokenization algorithm. +The {{stack-ml-features}} support transformer models that conform to the standard BERT model interface and use the WordPiece tokenization algorithm. {{stack-ml-features}} will always use the non-fast ("slow") tokenizer variant for all supported models. This ensures deterministic and stable tokenization results across different platforms and avoids potential differences in handling between fast and slow implementations. The current list of supported architectures is: From 2de8432b4fbacc889bb33ddf1af9df3721d3382f Mon Sep 17 00:00:00 2001 From: Pius Fung Date: Wed, 13 Aug 2025 15:39:01 -0700 Subject: [PATCH 2/3] Update explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md Co-authored-by: David Kyle --- explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md b/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md index 297b29cd68..a52dc18359 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md @@ -15,7 +15,7 @@ products: The minimum dedicated ML node size for deploying and using the {{nlp}} models is 16 GB in {{ech}} if [deployment autoscaling](../../../deploy-manage/autoscaling.md) is turned off. Turning on autoscaling is recommended because it allows your deployment to dynamically adjust resources based on demand. Better performance can be achieved by using more allocations or more threads per allocation, which requires bigger ML nodes. Autoscaling provides bigger nodes when required. If autoscaling is turned off, you must provide suitably sized nodes yourself. :::: -The {{stack-ml-features}} support transformer models that conform to the standard BERT model interface and use the WordPiece tokenization algorithm. {{stack-ml-features}} will always use the non-fast ("slow") tokenizer variant for all supported models. This ensures deterministic and stable tokenization results across different platforms and avoids potential differences in handling between fast and slow implementations. +The {{stack-ml-features}} support transformer models with the following architectures: The current list of supported architectures is: From 29ec13c360e98962d2d6bc07615902abff937f5d Mon Sep 17 00:00:00 2001 From: Pius Fung Date: Fri, 15 Aug 2025 11:20:08 -0700 Subject: [PATCH 3/3] Update explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md b/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md index a52dc18359..a049339e72 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md @@ -17,8 +17,6 @@ The minimum dedicated ML node size for deploying and using the {{nlp}} models is The {{stack-ml-features}} support transformer models with the following architectures: -The current list of supported architectures is: - * BERT * BART * DPR bi-encoders