You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This repository contains optimized Python implementations of the Fisher Scoring algorithm for various logistic regression models. With version 2.0, the core algorithms are now significantly faster due to optimized matrix operations and reduced memory usage, providing faster convergence for larger datasets.
10
10
11
11
```python
12
12
%pip install fisher-scoring
13
-
from fisher_scoring importFisherScoringLogisticRegression
This repository contains a Python package with scikit-learn compatible implementations of the Fisher Scoring algorithm for various logistic regression use cases:
28
+
This repository contains a Python package with scikit-learn compatible implementations of the Fisher Scoring algorithm for various modeling problems.
29
+
30
+
The packages provides implementations of logistic regression (MLE for binary, multiclass, and binary imbalanced) for proportions (risk or prevalence) and Poisson and Negative Binomial regression for log-linear regression for incidence rates.
3. Imbalanced classification problems: **Focal Loss Logistic Regression**.
35
+
4. Count modeling problems: **Poisson Regression** and **Negative Binomial Regression**.
33
36
34
37
### Fisher Scoring Algorithm
35
38
@@ -38,31 +41,33 @@ The Fisher Scoring algorithm is an iterative optimization technique that estimat
38
41
There are two types of information matrices used in the Fisher Scoring algorithm:
39
42
40
43
***Expected Information Matrix**: Relies on predicted probabilities, providing an efficient approximation for the information matrix.
41
-
***Observed Information Matrix**: Uses ground truth labels to calculate the information matrix, often resulting in more reliable inference metrics.
44
+
***Empirical Information Matrix**: Uses ground truth labels to calculate the information matrix, often resulting in more reliable inference metrics.
42
45
43
46
These information matrices are used to derive standard errors of estimates to calculate detailed model statistics, including Wald statistics, p-values, and confidence intervals at a chosen level.
44
47
48
+
Source: [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://arxiv.org/pdf/1905.12558).
49
+
45
50
### Implementation Notes
46
51
47
-
-**Fisher Scoring Multinomial Regression**
48
-
The `FisherScoringMultinomialRegression` model differs from standard statistical multinomial logistic regression by using all classes rather than $K - 1$. This approach allows multi-class classification problems to be converted to binary problems by calculating $1 - P_{Class=1}$.
52
+
-**Multinomial Logistic Regression**
53
+
The `MultinomialLogisticRegression` model differs from standard statistical multinomial logistic regression by using all classes rather than $K - 1$. This approach allows multi-class classification problems to be converted to binary problems by calculating $1 - P_{Class=1}$.
49
54
50
-
-**Fisher Scoring Focal Regression**
51
-
The `FisherScoringFocalRegression` class employs a non-standard focal log-likelihood function in its optimization process leveraging $\gamma$ to focus on difficult-to-classify examples.
55
+
-**Focal Loss Regression**
56
+
The `FocalLossRegression` class employs a non-standard focal log-likelihood function in its optimization process leveraging $\gamma$ to focus on difficult-to-classify examples.
52
57
The focal loss function, originally developed for object detection, prioritizes difficult-to-classify examples—often the minority class—by reducing the contribution of easy-to-classify samples. It introduces a focusing parameter, *gamma*, which down-weights the influence of easily classified instances, thereby concentrating learning on challenging cases.
53
58
54
59
Source: [Focal Loss for Dense Object Detection](https://arxiv.org/abs/1708.02002).
55
60
56
61
## Models
57
62
58
-
### Fisher Scoring Logistic Regression
63
+
### Logistic Regression
59
64
60
-
The `FisherScoringLogisticRegression` class is a custom implementation of logistic regression using the Fisher scoring algorithm. It provides methods for fitting the model, making predictions, and computing model statistics, including standard errors, Wald statistics, p-values, and confidence intervals.
65
+
The `LogisticRegression` class is a custom implementation of logistic regression using the Fisher scoring algorithm. It provides methods for fitting the model, making predictions, and computing model statistics, including standard errors, Wald statistics, p-values, and confidence intervals.
61
66
62
67
**Parameters:**
63
68
-`epsilon`: Convergence threshold for the algorithm.
64
69
-`max_iter`: Maximum number of iterations for the algorithm.
65
-
-`information`: Type of information matrix to use ('expected' or 'observed').
70
+
-`information`: Type of information matrix to use ('expected' or 'empirical').
66
71
-`use_bias`: Include a bias term in the model.
67
72
-`significance`: Significance level for computing confidence intervals.
68
73
@@ -76,14 +81,14 @@ The `FisherScoringLogisticRegression` class is a custom implementation of logist
76
81
-`summary()`: Get a summary of model parameters, standard errors, p-values, and confidence intervals.
77
82
-`display_summary()`: Display a summary of model parameters, standard errors, p-values, and confidence intervals.
78
83
79
-
### Fisher Scoring Multinomial Regression
84
+
### Multinomial Logistic Regression
80
85
81
-
The `FisherScoringMultinomialRegression` class implements the Fisher Scoring algorithm for multinomial logistic regression, suitable for multi-class classification tasks.
86
+
The `MultinomialLogisticRegression` class implements the Fisher Scoring algorithm for multinomial logistic regression, suitable for multi-class classification tasks.
82
87
83
88
**Parameters:**
84
89
-`epsilon`: Convergence threshold for the algorithm.
85
90
-`max_iter`: Maximum number of iterations for the algorithm.
86
-
-`information`: Type of information matrix to use ('expected' or 'observed').
91
+
-`information`: Type of information matrix to use ('expected' or 'empirical').
87
92
-`use_bias`: Include a bias term in the model.
88
93
-`significance`: Significance level for computing confidence intervals.
89
94
-`verbose`: Enable verbose output.
@@ -98,15 +103,15 @@ The `FisherScoringMultinomialRegression` class implements the Fisher Scoring alg
98
103
99
104
The algorithm is in a beta version and may require further testing and optimization to speed up matrix operations.
100
105
101
-
### Fisher Scoring Focal Loss Regression
106
+
### Focal Loss Regression
102
107
103
-
The `FisherScoringFocalRegression` class implements the Fisher Scoring algorithm with focal loss, designed for imbalanced classification problems where the positive class is rare.
108
+
The `FocalLossRegression` class implements the Fisher Scoring algorithm with focal loss, designed for imbalanced classification problems where the positive class is rare.
104
109
105
110
**Parameters:**
106
111
-`gamma`: Focusing parameter for focal loss.
107
112
-`epsilon`: Convergence threshold for the algorithm.
108
113
-`max_iter`: Maximum number of iterations for the algorithm.
109
-
-`information`: Type of information matrix to use ('expected' or 'observed').
114
+
-`information`: Type of information matrix to use ('expected' or 'empirical').
110
115
-`use_bias`: Include a bias term in the model.
111
116
-`verbose`: Enable verbose output.
112
117
@@ -120,33 +125,71 @@ The `FisherScoringFocalRegression` class implements the Fisher Scoring algorithm
120
125
-`summary()`: Get a summary of model parameters, standard errors, p-values, and confidence intervals.
121
126
-`display_summary()`: Display a summary of model parameters, standard errors, p-values, and confidence intervals.
122
127
123
-
##Installation
128
+
### Poisson Regression
124
129
125
-
To use the models, clone the repository and install the required dependencies.
130
+
The `PoissonRegression` class implements the Fisher Scoring algorithm for Poisson regression, suitable for modeling count data.
-`max_iter`: Maximum number of iterations for optimization.
134
+
-`epsilon`: Convergence tolerance.
135
+
-`use_bias`: Whether to include an intercept term.
136
+
137
+
**Methods:**
138
+
-`fit(X, y)`: Fit the model to the data.
139
+
-`predict(X)`: Predict mean values for the Poisson model.
140
+
-`calculate_st_errors(X)`: Calculate standard errors for the coefficients.
132
141
133
-
Alternatively, install the package directly from PyPI.
142
+
### Negative Binomial Regression
134
143
135
-
```bash
136
-
pip install fisher-scoring
137
-
```
144
+
The `NegativeBinomialRegression` class implements the Fisher Scoring algorithm for Negative Binomial regression, suitable for overdispersed count data.
145
+
146
+
**Parameters:**
147
+
-`max_iter`: Maximum number of iterations for optimization.
148
+
-`epsilon`: Convergence tolerance.
149
+
-`use_bias`: Whether to include an intercept term.
150
+
-`alpha`: Fixed dispersion parameter (overdispersion adjustment for Negative Binomial).
151
+
-`phi`: Constant scale parameter.
152
+
-`offset`: Offset term for the linear predictor.
153
+
154
+
**Methods:**
155
+
-`fit(X, y)`: Fit the model to the data.
156
+
-`predict(X)`: Predict mean values for the Negative Binomial model.
157
+
-`calculate_st_errors(X)`: Calculate standard errors for the coefficients.
158
+
159
+
## Utilities
160
+
161
+
### Visualization
162
+
163
+
The package includes a utility function for visualizing observed vs predicted probabilities for count data, which can be useful for users working with Poisson and Negative Binomial models.
164
+
165
+
**Function:**
166
+
-`plot_observed_vs_predicted(y, mu, max_count=15, alpha=None, title="Observed vs Predicted Probabilities", model_name="Model", ax=None, plot_params=None)`: Plot observed vs predicted probabilities for count data.
167
+
168
+
**Parameters:**
169
+
-`y`: Observed count data.
170
+
-`mu`: Predicted mean values from the model.
171
+
-`max_count`: Maximum count to consider for probabilities.
172
+
-`alpha`: Overdispersion parameter for Negative Binomial. If None, assumes Poisson (alpha=0).
173
+
-`title`: Title for the plot.
174
+
-`model_name`: Name of the model for labeling.
175
+
-`ax`: Matplotlib axis to plot on.
138
176
139
177
## Change Log
140
178
179
+
-**v2.0.4**
180
+
- Added a beta version of Poisson and Negative Binomial regression using Fisher Scoring.
181
+
- Changed naming conventions for simplicity and consistency.
182
+
- Changed poetry to uv for packaging.
183
+
141
184
-**v2.0.3**
142
185
- Added a new functionality of inference of mean responses with confidence intervals for all algorithms.
143
186
- Focal logistic regression now supports all model statistics, including standard errors, Wald statistics, p-values, and confidence intervals.
144
187
145
188
-**v2.0.2**
146
-
-**Bug Fixes**: Fixed the `FisherScoringMultinomialRegression` class to have flexible NumPy data types.
189
+
-**Bug Fixes**: Fixed the `MultinomialLogisticRegression` class to have flexible NumPy data types.
147
190
148
191
-**v2.0.1**
149
-
-**Bug Fixes**: Removed the debug print statement from the `FisherScoringLogisticRegression` class.
192
+
-**Bug Fixes**: Removed the debug print statement from the `LogisticRegression` class.
150
193
151
194
-**v2.0**
152
195
-**Performance Improvements**: Performance Enhancements: Optimized matrix calculations for substantial speed and memory efficiency improvements across all models. Leveraging streamlined operations, this version achieves up to 290x faster convergence. Performance gains per model:
0 commit comments