Nonparametric statistical tests for treatment heterogeneity and rank consistency across multiple ML algorithms
+ + + Source:vignettes/test_itr.Rmd
+ test_itr.RmdIn practice, machine learning (ML) algorithms may fail to ascertain
+heterogeneous treatment effects due to small small sample sizes, high
+dimensionality, and arbitrary parameter-tuning. The
+test_itr function allows users to empirically validate the
+estimates of GATEs under various ML algorithms with statistical testing.
+In particular, there are two types of nonparametric statistical tests:
+(1) test for across group treatment effects heterogeneity and (2) test
+of rank consistency of GATEs. The following provides a description of
+each test. The tests are based on the idea that, if an ML algorithm
+produces a reasonable scoring rule (achieved by the
+estimate_itr function), it is reasonable to expect that (1)
+the GATEs across groups are heterogeneous; and (2) the rank ordering of
+the GATEs based on their magnitude should be mononotic.
Following the previous examples, we first estimate GATEs using causal
+forest (causal forest), Bayesian Additive Regression Trees
+(bartc), LASSO (lasso), random forest
+(rf) under cross-validation using the
+estimate_itr function. We specify the number of groups to
+divide the sample into through the ngates argument. By
+setting ngates = 5 in the example below, we estimate the
+heterogeneous impact of small class sizes on students’ writing scores
+across 5 groups of students.
+# library(evalITR)
+devtools::load_all(".")
+
+# specify the trainControl method
+fitControl <- caret::trainControl(
+ method = "repeatedcv",
+ number = 2,
+ repeats = 2)
+# estimate ITR
+set.seed(2023)
+fit_cv <- estimate_itr(
+ treatment = "treatment",
+ form = user_formula,
+ data = star_data,
+ trControl = fitControl,
+ algorithms = c(
+ "causal_forest", # from caret
+ "bartc", # from caret
+ "lasso", # from caret
+ "rf"), # from caret
+ budget = 0.2, # 20% budget constraint
+ n_folds = 5, # 5-fold cross-validation
+ ngates = 5) # 5 groups
+#> Evaluate ITR with cross-validation ...
+#> fitting treatment model via method 'bart'
+#> fitting response model via method 'bart'
+#> fitting treatment model via method 'bart'
+#> fitting response model via method 'bart'
+#> fitting treatment model via method 'bart'
+#> fitting response model via method 'bart'
+#> fitting treatment model via method 'bart'
+#> fitting response model via method 'bart'
+#> fitting treatment model via method 'bart'
+#> fitting response model via method 'bart'
+
+# evaluate ITR
+est_cv <- evaluate_itr(fit_cv)
+
+# extract GATEs estimates
+summary(est_cv)$GATE
+#> # A tibble: 20 × 8
+#> estimate std.deviation algorithm group statistic p.value upper lower
+#> <dbl> <dbl> <chr> <int> <dbl> <dbl> <dbl> <dbl>
+#> 1 -52.3 125. causal_forest 1 -0.417 0.677 193. -298.
+#> 2 -61.8 109. causal_forest 2 -0.568 0.570 151. -275.
+#> 3 22.8 82.6 causal_forest 3 0.275 0.783 185. -139.
+#> 4 135. 120. causal_forest 4 1.13 0.259 369. -99.3
+#> 5 -24.9 124. causal_forest 5 -0.202 0.840 217. -267.
+#> 6 -24.1 88.2 bartc 1 -0.273 0.785 149. -197.
+#> 7 -25.5 59.5 bartc 2 -0.429 0.668 91.1 -142.
+#> 8 -46.9 101. bartc 3 -0.465 0.642 151. -245.
+#> 9 131. 102. bartc 4 1.28 0.199 330. -68.7
+#> 10 -15.1 116. bartc 5 -0.130 0.896 212. -242.
+#> 11 -56.5 92.0 lasso 1 -0.614 0.539 124. -237.
+#> 12 98.6 59.5 lasso 2 1.66 0.0975 215. -18.0
+#> 13 -25.6 114. lasso 3 -0.224 0.823 199. -250.
+#> 14 40.4 117. lasso 4 0.344 0.731 271. -190.
+#> 15 -38.0 96.8 lasso 5 -0.393 0.695 152. -228.
+#> 16 -117. 58.7 rf 1 -1.99 0.0469 -1.61 -232.
+#> 17 2.78 124. rf 2 0.0224 0.982 246. -240.
+#> 18 22.7 111. rf 3 0.204 0.838 241. -196.
+#> 19 145. 94.6 rf 4 1.53 0.125 331. -40.3
+#> 20 -35.0 118. rf 5 -0.296 0.767 197. -267.The table reports the quintile GATEs (\(K = +5\)) estimates for each ML algorithm. We find that the Random +Forest is able to produce statistically negative GATE for the lowest +quantitle group (group 1) under cross-validation. This provides evidence +that the Random Forest is able to identify a 20% subgroup whose writing +scores are negatively impacted by small class sizes.
+We now conduct the statistical tests of treatment effect
+heterogeneity and rank consistency to validate these GATEs estimates. We
+use the module object output test_est_cv from the
+evaluate_itr function as the input object for the
+test_itr function to conduct 2 tests simultaneously. We can
+summarize the test statistics and the p-values using the
+summary function. Lastly, we use the nsims
+argument to specify the number of simulations to conduct for each test.
+The default is 1000 simulations.
+# conduct nonparametric tests
+test_est_cv <- test_itr(est_cv,
+ nsim = 5000)
+#> Conduct hypothesis tests for GATEs unde cross-validation ...
+
+# summarize test statistics and p-values
+summary(test_est_cv)
+#> ── The Consistency Test Results for GATEs ──────────────────────────────────────
+#> No consistency results available (sample-splitting).
+#>
+#> ── The Heterogeneity Test Results for GATEs ────────────────────────────────────
+#> No heterogeneity results available (sample-splitting).
+#>
+#> ── The Consistency Test Results for GATEs (Cross-validation) ───────────────────
+#> algorithm statistic p.value
+#> 1 causal_forest 0.83 0.74
+#> 2 bartc 1.03 0.63
+#> 3 lasso 0.24 0.82
+#> 4 rf 1.32 0.66
+#>
+#> ── The Heterogeneity Test Results for GATEs (Cross-validation) ─────────────────
+#> algorithm statistic p.value
+#> 1 causal_forest 1.9 0.86
+#> 2 bartc 2.3 0.81
+#> 3 lasso 3.3 0.65
+#> 4 rf 6.6 0.25The table reports the resulting values of test statistics and the +p-values for each test under each algorithm. We find that none of the ML +algorithms is able to reject the treatment effect homogeneity hypothesis +under cross-validation, which indicates that these algorithms failed to +identify statistically significant GATEs estimate for any subgroup. In +addition, none of the ML algorithms is able to reject the rank +consistency hypothesis under cross-validation. Thus, there is no strong +statistical evidence that these algorithms are producing unreliable +GATEs.
+