sunab() inconsistencies

[ocolluphid]

Inconsistent Results with sunab() Function

Hi fixest team!

Thank you for the highly performant implementation of Sun-Abraham via the sunab() function! However, I'm encountering inconsistent results where regression output sometimes contains NAs despite no detected collinearity issues, and sometimes rerunning the same regression and data occasionally yields complete results without nulls (and in both cases the R2 and RMSE match, despite the null coefficients.

Potential Sources of Inconsistency

I suspect the following factors might be causing these inconsistencies. It would be helpful to clarify these points in the sunab() documentation:

1. Fixed Effects Time Period Matching

As mentioned in this thread with Kyle, we need FEs on top of sunab(). I'm using year FEs while the cohort and current periods in sunab() are expressed in quarterly units. @kylebutts

Question: Does the FE time period have to match the time units in sunab()?

2. Period and Cohort Format Compatibility

When expressing period as a relative period (RP), can I pass cohort as a string format like "2022Q1" instead of a date format like "01-01-2022"?

3. Never-Treated Group Detection

I'm concerned that cases with NAs might result from the function not automatically detecting the "never treated" group, even though I assign them a cohort date years/quarters before the date range of interest. I'm attempting to explicitly specify the "never treated" date using ref.c.

Questions:

• Would passing the never-treated reference as a string cause problems?
• Are there examples in the documentation showing proper ref.c usage?
• Does specifying ref.c improve consistency between runs by avoiding imputation of the "never treated" group?

4. Never-Treated Group Timing

In the documentation example, 1000 is set as the year treated for the "never treated" group. Does it matter whether we set the never-treated year outside the range to the future or to the past?

Request

In addition to addressing these questions here, does it also make sense to address them in the documentation via examples? Thanks!

[kylebutts]

I'm encountering inconsistent results where regression output sometimes contains NAs

Could you create a reprex with this and I can try and see what is going on

[ocolluphid]

here is a case is just spotted while re-running my code

NA case:

sunab20 = feols(totalCount_avg ~ sunab(treat_event_quarter_date, current_quarter_date) | rcid + current_quarter_date, data = final_subset)
NOTE: 1 observation removed because of NA values (RHS: 1).
print(sunab20)
OLS estimation, Dep. Var.: totalCount_avg
Observations: 87,429
Fixed-effects: rcid: 2,681,  current_quarter_date: 40
Standard-errors: Clustered (rcid) 
                         Estimate Std. Error t value Pr(>|t|) 
current_quarter_date::-4       NA         NA      NA       NA 
current_quarter_date::-3       NA         NA      NA       NA 
current_quarter_date::-2       NA         NA      NA       NA 
current_quarter_date::0        NA         NA      NA       NA 
current_quarter_date::1        NA         NA      NA       NA 
current_quarter_date::2        NA         NA      NA       NA 
current_quarter_date::3        NA         NA      NA       NA 
current_quarter_date::4        NA         NA      NA       NA 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
RMSE: 84.7     Adj. R2: 0.877117
             Within R2: 0.020842

Non-NA case. Same model run a few seconds later:

> sunab20_quarterFE = feols(totalCount_avg ~ sunab(treat_event_quarter_date, current_quarter_date) | rcid + current_quarter_date, data = final_subset)
NOTE: 1 observation removed because of NA values (RHS: 1).
> print(sunab20_quarterFE)
OLS estimation, Dep. Var.: totalCount_avg
Observations: 87,429
Fixed-effects: rcid: 2,681,  current_quarter_date: 40
Standard-errors: Clustered (rcid) 
                         Estimate Std. Error  t value   Pr(>|t|)    
current_quarter_date::-4 -26.8737    3.75024 -7.16587 9.9477e-13 ***
current_quarter_date::-3 -19.5481    2.76546 -7.06864 1.9873e-12 ***
current_quarter_date::-2 -10.4310    1.48561 -7.02137 2.7735e-12 ***
current_quarter_date::0   13.8497    1.94625  7.11611 1.4191e-12 ***
current_quarter_date::1   30.5304    3.99030  7.65116 2.7590e-14 ***
current_quarter_date::2   47.4711    6.72552  7.05836 2.1371e-12 ***
current_quarter_date::3   64.9541    9.40131  6.90905 6.0727e-12 ***
current_quarter_date::4   78.7742   11.30803  6.96621 4.0812e-12 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
RMSE: 84.7     Adj. R2: 0.877117
             Within R2: 0.020842

here are some freqs for understanding the period variables:

freq(final_subset$current_quarter_date)
              n   % val%
2015-01-01 2075 2.4  2.4
2015-04-01 2087 2.4  2.4
2015-07-01 2099 2.4  2.4
2015-10-01 2106 2.4  2.4
2016-01-01 2110 2.4  2.4
2016-04-01 2121 2.4  2.4
2016-07-01 2126 2.4  2.4
2016-10-01 2138 2.4  2.4
2017-01-01 2140 2.4  2.4
2017-04-01 2145 2.5  2.5
2017-07-01 2145 2.5  2.5
2017-10-01 2153 2.5  2.5
2018-01-01 2156 2.5  2.5
2018-04-01 2164 2.5  2.5
2018-07-01 2172 2.5  2.5
2018-10-01 2173 2.5  2.5
2019-01-01 2174 2.5  2.5
2019-04-01 2175 2.5  2.5
2019-07-01 2181 2.5  2.5
2019-10-01 2187 2.5  2.5
2020-01-01 2194 2.5  2.5
2020-04-01 2211 2.5  2.5
2020-07-01 2237 2.6  2.6
2020-10-01 2265 2.6  2.6
2021-01-01 2294 2.6  2.6
2021-04-01 2319 2.7  2.7
2021-07-01 2324 2.7  2.7
2021-10-01 2325 2.7  2.7
2022-01-01 2321 2.7  2.7
2022-04-01 2304 2.6  2.6
2022-07-01 2291 2.6  2.6
2022-10-01 2261 2.6  2.6
2023-01-01 2223 2.5  2.5
2023-04-01 2195 2.5  2.5
2023-07-01 2164 2.5  2.5
2023-10-01 2163 2.5  2.5
2024-01-01 2147 2.5  2.5
2024-04-01 2133 2.4  2.4
2024-07-01 2123 2.4  2.4
2024-10-01 2109 2.4  2.4

freq(final_subset$treat_event_quarter_date)
               n    % val%
2004-10-01 81920 93.7 93.7
2015-01-01    10  0.0  0.0
2015-04-01    48  0.1  0.1
2015-07-01    28  0.0  0.0
2015-10-01    72  0.1  0.1
2016-01-01    36  0.0  0.0
2016-04-01   108  0.1  0.1
2016-07-01   108  0.1  0.1
2016-10-01    63  0.1  0.1
2017-01-01    36  0.0  0.0
2017-04-01   117  0.1  0.1
2017-07-01   117  0.1  0.1
2017-10-01   144  0.2  0.2
2018-01-01    99  0.1  0.1
2018-04-01    81  0.1  0.1
2018-07-01   108  0.1  0.1
2018-10-01   180  0.2  0.2
2019-01-01    90  0.1  0.1
2019-04-01   108  0.1  0.1
2019-07-01   189  0.2  0.2
2019-10-01   126  0.1  0.1
2020-01-01   153  0.2  0.2
2020-04-01   108  0.1  0.1
2020-07-01   135  0.2  0.2
2020-10-01   162  0.2  0.2
2021-01-01   243  0.3  0.3
2021-04-01   243  0.3  0.3
2021-07-01   342  0.4  0.4
2021-10-01   441  0.5  0.5
2022-01-01   387  0.4  0.4
2022-04-01   378  0.4  0.4
2022-07-01   153  0.2  0.2
2022-10-01   144  0.2  0.2
2023-01-01   126  0.1  0.1
2023-04-01    90  0.1  0.1
2023-07-01   126  0.1  0.1
2023-10-01    72  0.1  0.1
2024-01-01    88  0.1  0.1
2024-04-01   105  0.1  0.1
2024-07-01    66  0.1  0.1
2024-10-01    80  0.1  0.1

I think it's the unpredictability of when NAs will arise that's throwing me off. Maybe I should set the never treated group to the year 2050 instead of the year 2004? I have seen similar issue arise when using year_FEs but with quarters still used within sunab()

[kylebutts]

Are you saying that you run the same exact code with the same exact data and it's producing NAs in one and estimates in the other? If so, that's very not good.

The reference date is fine; just use ref.c with the appropriate date. For quarterly data, try using yearquarter which will convert your date variable to a decimal. I think that could be causing the issue with Date variables being used

[ocolluphid]

that's right, and the puzzling part is that R2 stays exactly the same which suggests to me that the FEs are being estimated but not shown?
thanks, I will try implementing that and running it across multiple specifications to see if it becomes more consistent

[kylebutts]

That's really bizarre; it might be some numerical precision issues.

If you can easily share the data, could you email it to me buttskyle96@gmail.com and I will see if I can replicate this?

[lrberge]

Hi @ocolluphid: could you rerun with the current version?
There have been important internal changes in v0.13 and this is very likely that it solves the issue you raised.