Skip to content

Spark to scikit gradient boost classifier #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
pavan02 wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Spark to scikit gradient boost classifier #26

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added python/spark-to-scikit/RandomForestClassifierXGBoost/HIGGS_100.csv/._SUCCESS.crc
View file
Open in desktop
Binary file not shown.
Binary file added ...sifierXGBoost/HIGGS_100.csv/.part-00000-758749fa-cbbd-473e-b1cd-4d06fd714edd-c000.csv.crc
View file
Open in desktop
Binary file not shown.
Empty file added python/spark-to-scikit/RandomForestClassifierXGBoost/HIGGS_100.csv/_SUCCESS
View file
Open in desktop
Empty file.
100 changes: 100 additions & 0 deletions ...tClassifierXGBoost/HIGGS_100.csv/part-00000-758749fa-cbbd-473e-b1cd-4d06fd714edd-c000.csv
View file
Open in desktop

Large diffs are not rendered by default.

869 changes: 869 additions & 0 deletions python/spark-to-scikit/RandomForestClassifierXGBoost/converted_scikit_model.json
View file
Open in desktop

Large diffs are not rendered by default.

78 changes: 78 additions & 0 deletions python/spark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
0.5 0.5
GBTClassificationModel: uid = GBTClassifier_6cdf4f6cf4e7, numTrees=2, numClasses=2, numFeatures=28
Tree 0 (weight 1.0):
If (feature 25 <= 1.5447615385055542)
If (feature 26 <= 0.7740358114242554)
If (feature 18 <= -0.22865427285432816)
Predict: -1.0
Else (feature 18 > -0.22865427285432816)
If (feature 3 <= 1.1699432134628296)
Predict: 1.0
Else (feature 3 > 1.1699432134628296)
Predict: -1.0
Else (feature 26 > 0.7740358114242554)
If (feature 3 <= 1.975119411945343)
If (feature 23 <= 0.9756079912185669)
Predict: -1.0
Else (feature 23 > 0.9756079912185669)
If (feature 7 <= -1.712132751941681)
Predict: 0.0
Else (feature 7 > -1.712132751941681)
Predict: 0.9393939393939394
Else (feature 3 > 1.975119411945343)
If (feature 6 <= 0.2717926800251007)
Predict: -1.0
Else (feature 6 > 0.2717926800251007)
Predict: 1.0
Else (feature 25 > 1.5447615385055542)
If (feature 17 <= 1.5525725483894348)
If (feature 2 <= -1.5533004403114319)
If (feature 0 <= 1.7702469229698181)
Predict: 1.0
Else (feature 0 > 1.7702469229698181)
Predict: -1.0
Else (feature 2 > -1.5533004403114319)
Predict: -1.0
Else (feature 17 > 1.5525725483894348)
If (feature 0 <= 0.5344780385494232)
Predict: -1.0
Else (feature 0 > 0.5344780385494232)
Predict: 1.0
Tree 1 (weight 0.1):
If (feature 9 <= 0.33968763053417206)
If (feature 0 <= 0.6849115490913391)
Predict: -3.46988735068147
Else (feature 0 > 0.6849115490913391)
Predict: 0.53011264931853
Else (feature 9 > 0.33968763053417206)
If (feature 25 <= 1.2138408422470093)
If (feature 26 <= 0.7740358114242554)
If (feature 18 <= -0.22865427285432816)
Predict: -0.4768116880884702
Else (feature 18 > -0.22865427285432816)
If (feature 3 <= 1.1699432134628296)
Predict: 0.4768116880884702
Else (feature 3 > 1.1699432134628296)
Predict: -0.4768116880884703
Else (feature 26 > 0.7740358114242554)
If (feature 7 <= -1.712132751941681)
Predict: 2.0
Else (feature 7 > -1.712132751941681)
If (feature 3 <= 1.975119411945343)
Predict: 0.5301126493185299
Else (feature 3 > 1.975119411945343)
Predict: -0.09536233761769389
Else (feature 25 > 1.2138408422470093)
If (feature 7 <= -1.712132751941681)
Predict: -2.0
Else (feature 7 > -1.712132751941681)
If (feature 11 <= 0.019821979105472565)
If (feature 25 <= 1.6286334991455078)
Predict: 0.5148838032527986
Else (feature 25 > 1.6286334991455078)
Predict: -0.23840584404423515
Else (feature 11 > 0.019821979105472565)
If (feature 13 <= 1.5283446907997131)
Predict: -0.4768116880884702
Else (feature 13 > 1.5283446907997131)
Predict: 0.17670421643950984
Binary file added python/spark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model/data/._SUCCESS.crc
View file
Open in desktop
Binary file not shown.
Binary file added ...spark_model/data/.part-00000-5594e26e-c145-43ff-a90a-ccee363afaca-c000.snappy.parquet.crc
View file
Open in desktop
Binary file not shown.
Empty file added python/spark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model/data/_SUCCESS
View file
Open in desktop
Empty file.
Binary file added ...iggs_spark_model/data/part-00000-5594e26e-c145-43ff-a90a-ccee363afaca-c000.snappy.parquet
View file
Open in desktop
Binary file not shown.
Binary file added ...on/spark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model/metadata/._SUCCESS.crc
View file
Open in desktop
Binary file not shown.
Binary file added .../spark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
Empty file added python/spark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model/metadata/_SUCCESS
View file
Open in desktop
Empty file.
1 change: 1 addition & 0 deletions python/spark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"class":"org.apache.spark.ml.classification.GBTClassificationModel","timestamp":1652667981709,"sparkVersion":"3.2.1","uid":"GBTClassifier_6cdf4f6cf4e7","paramMap":{"featuresCol":"features","maxIter":2,"labelCol":"label","thresholds":[0.5,0.5]},"defaultParamMap":{"subsamplingRate":1.0,"stepSize":0.1,"cacheNodeIds":false,"maxBins":32,"featuresCol":"features","maxMemoryInMB":256,"probabilityCol":"probability","maxIter":20,"lossType":"logistic","validationTol":0.01,"minWeightFractionPerNode":0.0,"minInfoGain":0.0,"rawPredictionCol":"rawPrediction","impurity":"variance","leafCol":"","featureSubsetStrategy":"all","predictionCol":"prediction","seed":7266179102098935354,"checkpointInterval":10,"labelCol":"label","maxDepth":5,"minInstancesPerNode":1},"numFeatures":28,"numTrees":2}
Binary file added ...ark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model/treesMetadata/._SUCCESS.crc
View file
Open in desktop
Binary file not shown.
Binary file added ...el/treesMetadata/.part-00000-59a288da-31d0-4bbd-b097-b4f6c77a77ac-c000.snappy.parquet.crc
View file
Open in desktop
Binary file not shown.
Empty file added ...on/spark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model/treesMetadata/_SUCCESS
View file
Open in desktop
Empty file.
Binary file added ...k_model/treesMetadata/part-00000-59a288da-31d0-4bbd-b097-b4f6c77a77ac-c000.snappy.parquet
View file
Open in desktop
Binary file not shown.
79 changes: 79 additions & 0 deletions python/spark-to-scikit/RandomForestClassifierXGBoost/higgs_spark_model_train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import StructType, StructField, DoubleType
from pyspark.ml.classification import GBTClassifier
from pyspark.ml.feature import VectorAssembler

def save_model(model_string, filename):
with open(filename, "w") as text_file:
text_file.write(model_string)

if __name__ == '__main__':
sc = SparkContext()
sqlContext = SQLContext(sc)

schema = StructType([ \
StructField("label", DoubleType(),True), \
StructField("f0", DoubleType(),True), \
StructField("f1", DoubleType(),True), \
StructField("f2", DoubleType(),True), \
StructField("f3", DoubleType(), True), \
StructField("f4", DoubleType(), True), \
StructField("f5", DoubleType(), True), \
StructField("f6", DoubleType(), True), \
StructField("f7", DoubleType(), True), \
StructField("f8", DoubleType(), True), \
StructField("f9", DoubleType(), True), \
StructField("f10", DoubleType(), True), \
StructField("f11", DoubleType(),True), \
StructField("f12", DoubleType(),True), \
StructField("f13", DoubleType(),True), \
StructField("f14", DoubleType(), True), \
StructField("f15", DoubleType(), True), \
StructField("f16", DoubleType(), True), \
StructField("f17", DoubleType(), True), \
StructField("f18", DoubleType(), True), \
StructField("f19", DoubleType(), True), \
StructField("f20", DoubleType(), True), \
StructField("f21", DoubleType(),True), \
StructField("f22", DoubleType(),True), \
StructField("f23", DoubleType(),True), \
StructField("f24", DoubleType(), True), \
StructField("f25", DoubleType(), True), \
StructField("f26", DoubleType(), True), \
StructField("f27", DoubleType(), True), \
])

df = sqlContext.read.format('com.databricks.spark.csv').schema(schema).load("HIGGS_100.csv/part-00000-758749fa-cbbd-473e-b1cd-4d06fd714edd-c000.csv").limit(100)
features_column = 'features'
label_column = 'label'
feature_cols = ['f0', 'f1', 'f2', 'f3','f4','f5',
'f6','f7','f8','f9','f10',
'f11','f12','f13','f14','f15',
'f16','f17','f18','f19','f20',
'f21', 'f22', 'f23', 'f24', 'f25',
'f26', 'f27']
assembler = VectorAssembler(inputCols=feature_cols, outputCol=features_column)
df = assembler.transform(df)

rf = GBTClassifier(featuresCol=features_column, labelCol=label_column, maxIter=2)
train, test = df.randomSplit([0.7, 0.3], seed = 2018)
rf_model = rf.fit(train)

# export model as string
model_string = rf_model.toDebugString
rf_model.setThresholds([0.5, 0.5])
thresholds = rf_model.getThresholds()
thresholds_string = " ".join(str(x) for x in thresholds)
save_model(thresholds_string + '\n' + model_string, 'higgs_spark_model.txt')

# save model using spark
rf_model.write().overwrite().save('higgs_spark_model')

# print predictions
predictions = rf_model.transform(test)
predictions.select(label_column, "prediction", 'probability').show()

# # save test rows as csv for validation
# test = test.drop(features_column)
# test.coalesce(1).write.mode("overwrite").option("header","true").csv("testdata_higgs_spark_model")
76 changes: 76 additions & 0 deletions python/spark-to-scikit/RandomForestClassifierXGBoost/metadata/higgs_scikit_model.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
|--- feature_25 <= 1.54
| |--- feature_26 <= 0.77
| | |--- feature_18 <= -0.23
| | | |--- value: [-1.00]
| | |--- feature_18 > -0.23
| | | |--- feature_3 <= 1.17
| | | | |--- value: [1.00]
| | | |--- feature_3 > 1.17
| | | | |--- value: [-1.00]
| |--- feature_26 > 0.77
| | |--- feature_3 <= 1.98
| | | |--- feature_23 <= 0.98
| | | | |--- value: [-1.00]
| | | |--- feature_23 > 0.98
| | | | |--- feature_7 <= -1.71
| | | | | |--- value: [0.00]
| | | | |--- feature_7 > -1.71
| | | | | |--- value: [0.94]
| | |--- feature_3 > 1.98
| | | |--- feature_6 <= 0.27
| | | | |--- value: [-1.00]
| | | |--- feature_6 > 0.27
| | | | |--- value: [1.00]
|--- feature_25 > 1.54
| |--- feature_17 <= 1.55
| | |--- feature_2 <= -1.55
| | | |--- feature_0 <= 1.77
| | | | |--- value: [1.00]
| | | |--- feature_0 > 1.77
| | | | |--- value: [-1.00]
| | |--- feature_2 > -1.55
| | | |--- value: [-1.00]
| |--- feature_17 > 1.55
| | |--- feature_0 <= 0.53
| | | |--- value: [-1.00]
| | |--- feature_0 > 0.53
| | | |--- value: [1.00]

|--- feature_9 <= 0.34
| |--- feature_0 <= 0.68
| | |--- value: [-3.47]
| |--- feature_0 > 0.68
| | |--- value: [0.53]
|--- feature_9 > 0.34
| |--- feature_25 <= 1.21
| | |--- feature_26 <= 0.77
| | | |--- feature_18 <= -0.23
| | | | |--- value: [-0.48]
| | | |--- feature_18 > -0.23
| | | | |--- feature_3 <= 1.17
| | | | | |--- value: [0.48]
| | | | |--- feature_3 > 1.17
| | | | | |--- value: [-0.48]
| | |--- feature_26 > 0.77
| | | |--- feature_7 <= -1.71
| | | | |--- value: [2.00]
| | | |--- feature_7 > -1.71
| | | | |--- feature_3 <= 1.98
| | | | | |--- value: [0.53]
| | | | |--- feature_3 > 1.98
| | | | | |--- value: [-0.10]
| |--- feature_25 > 1.21
| | |--- feature_7 <= -1.71
| | | |--- value: [-2.00]
| | |--- feature_7 > -1.71
| | | |--- feature_11 <= 0.02
| | | | |--- feature_25 <= 1.63
| | | | | |--- value: [0.51]
| | | | |--- feature_25 > 1.63
| | | | | |--- value: [-0.24]
| | | |--- feature_11 > 0.02
| | | | |--- feature_13 <= 1.53
| | | | | |--- value: [-0.48]
| | | | |--- feature_13 > 1.53
| | | | | |--- value: [0.18]

Binary file added ...stClassifierXGBoost/metadata/scikit_trees_visualization/scikit_tree_image_0.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added ...stClassifierXGBoost/metadata/scikit_trees_visualization/scikit_tree_image_1.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading