features.md

Apache Spark ML

Examples: main.py

• Feature extractors, transformers and selectors:
  • feature.Binarizer
  • feature.Bucketizer
  • feature.ChiSqSelectorModel (the result of fitting a feature.ChiSqSelector)
  • feature.ColumnPruner
  • feature.CountVectorizerModel (the result of fitting a feature.CountVectorizer)
  • feature.ElementwiseProduct
  • feature.IDFModel (the result of fitting a feature.IDF)
  • feature.ImputerModel (the result of fitting a feature.Imputer)
  • feature.IndexToString
  • feature.Interaction
  • feature.MaxAbsScalerModel (the result of fitting a feature.MaxAbsScaler)
  • feature.MinMaxScalerModel (the result of fitting a feature.MinMaxScaler)
  • feature.NGram
  • feature.OneHotEncoderModel (the result of fitting a feature.OneHotEncoder)
  • feature.PCAModel (the result of fitting a feature.PCA)
  • feature.PolynomialExpansion
  • feature.QuantileDiscretizer
  • feature.RegexTokenizer
  • feature.RFormulaModel (the result of fitting a feature.RFormula)
  • feature.RobustScalerModel (the result of fitting a feature.RobustScaler)
  • feature.SQLTransformer
    • Subqueries.
    • Control flow expressions case when and if.
    • Arithmetic operators +, -, * and /.
    • Comparison operators <, <=, ==, >= and >.
    • Logical operators and, or and not.
    • Math functions abs, ceil, exp, expm1, floor, hypot, ln, log10, log1p, pow and rint.
    • Trigonometric functions sin, asin, sinh, cos, acos, cosh, tan, atan, tanh.
    • Aggregation functions greatest and least.
    • RegExp functions regexp_replace and rlike.
    • String functions char_length, character_length, concat, lcase, length, lower, replace, substring, trim, ucase and upper.
    • Type cast functions boolean, cast, double, int and string.
    • Value functions in, isnan, isnull, isnotnull, negative and positive.
  • feature.StandardScalerModel (the result of fitting a feature.StandardScaler)
  • feature.StopWordsRemover
  • feature.StringIndexerModel (the result of fitting a feature.StringIndexer)
  • feature.Tokenizer
  • feature.VectorAssembler
  • feature.VectorAttributeRewriter
  • feature.VectorIndexerModel (the result of fitting a feature.VectorIndexer)
  • feature.VectorSizeHint
  • feature.VectorSlicer
• Prediction models:
  • classification.DecisionTreeClassificationModel
  • classification.GBTClassificationModel
  • classification.LinearSVCModel
  • classification.LogisticRegressionModel
  • classification.MultilayerPerceptronClassificationModel
  • classification.NaiveBayesModel
  • classification.RandomForestClassificationModel
  • clustering.KMeansModel
  • fpm.FPGrowthModel
  • regression.DecisionTreeRegressionModel
  • regression.GBTRegressionModel
  • regression.GeneralizedLinearRegressionModel
  • regression.IsotonicRegressionModel
  • regression.LinearRegressionModel
  • regression.RandomForestRegressionModel
• Prediction model chains:
  • PipelineModel
  • Referencing the prediction column (HasPredictionCol#getPredictionCol()) of earlier clustering, classification and regression models.
  • Referencing the predicted probabilities column (HasProbabilityCol#getProbabilityCol()) of earlier classification models.
• Hyperparameter selectors and tuners:
  • tuning.CrossValidatorModel
  • tuning.TrainValidationSplitModel

JPMML-SparkML

• Feature decorators and transformers:
  • org.jpmml.sparkml.feature.CategoricalDomainModel (the result of fitting a feature.CategoricalDomain)
  • org.jpmml.sparkml.feature.ContinuousDomainModel (the result of fitting a feature.ContinuousDomain)
  • org.jpmml.sparkml.feature.InvalidCategoryTransformer
  • org.jpmml.sparkml.feature.VectorDensifier (formerly org.jpmml.sparkml.feature.SparseToDenseTransformer)
  • org.jpmml.sparkml.feature.VectorDisassembler

LightGBM

Examples: main.scala

• Prediction models:
  • com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassificationModel
  • com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressionModel

XGBoost

Examples: main.scala

• Prediction models:
  • ml.dmlc.xgboost4j.scala.spark.XGBoostClassificationModel
  • ml.dmlc.xgboost4j.scala.spark.XGBoostRegressionModel