v2.0.5

Author: xRiskLab

.gitignore

@@ -1,6 +1,7 @@
 .DS_Store
 **/.DS_Store/
 .pytest_cache/
+.ruff_cache/
 **/.pytest_cache/
 __pycache__/
 **/__pycache__/
@@ -12,4 +13,5 @@ __pycache__/
 
 # Hide egg-info
 *.egg-info/
-dist/
+dist/
+.env

README.md

@@ -1,7 +1,7 @@
 # Fisher Scoring with Python
 
 **Author:** [xRiskLab](https://github.com/xRiskLab)<br>
-**Version:** v2.0.4<br>
+**Version:** v2.0.5<br>
 **License:** [MIT License](https://opensource.org/licenses/MIT) (2025)
 
 ![Title](https://github.com/xRiskLab/fisher-scoring/raw/main/docs/images/title.png)
@@ -10,15 +10,27 @@ This repository contains optimized Python implementations of the Fisher Scoring
 
 ```python
 %pip install fisher-scoring
-from fisher_scoring import LogisticRegression
+from fisher_scoring import LogisticRegression, RobustLogisticRegression, PoissonRegression
 
-# Initialize and fit model
+# Binary Classification
 model = LogisticRegression()
 model.fit(X_train, y_train)
-
-# Make predictions
 predictions = model.predict(X_test)
-probabilities = model.predict_proba(X_test)
+model.display_summary()  # Rich formatted output
+
+# Robust Classification (outlier-resistant)
+robust_model = RobustLogisticRegression(epsilon_contamination=0.05)
+robust_model.fit(X_train_contaminated, y_train_contaminated)
+robust_model.display_summary()  # Rich formatted output with robustness metrics
+
+# Count Data with Rate Modeling  
+import numpy as np
+exposure_times = np.random.uniform(0.5, 3.0, len(y_train))
+offset = np.log(exposure_times)  # Log exposure for rate modeling
+
+poisson_model = PoissonRegression(offset=offset, information="empirical")
+poisson_model.fit(X_train, y_train)
+poisson_model.display_summary()  # Rich formatted output
 ```
 
 ## Overview
@@ -27,12 +39,13 @@ probabilities = model.predict_proba(X_test)
 
 This repository contains a Python package with scikit-learn compatible implementations of the Fisher Scoring algorithm for various modeling problems.
 
-The packages provides implementations of logistic regression (MLE for binary, multiclass, and binary imbalanced) for proportions (risk or prevalence) and Poisson and Negative Binomial regression for log-linear regression for incidence rates.
+The packages provides implementations of logistic regression (MLE for binary, multiclass, and binary imbalanced) for proportions (risk or prevalence), robust logistic regression for outlier-resistant classification, and Poisson and Negative Binomial regression for log-linear regression for incidence rates.
 
 1. Binary classification problems: **Logistic Regression**.
-2. Multi-class classification problems: **Multinomial Logistic Regression**.
-3. Imbalanced classification problems: **Focal Loss Logistic Regression**.
-4. Count modeling problems: **Poisson Regression** and **Negative Binomial Regression**.
+2. Robust binary classification problems: **Robust Logistic Regression**.
+3. Multi-class classification problems: **Multinomial Logistic Regression**.
+4. Imbalanced classification problems: **Focal Loss Logistic Regression**.
+5. Count modeling problems: **Poisson Regression** and **Negative Binomial Regression**.
 
 ### Fisher Scoring Algorithm
 
@@ -81,6 +94,36 @@ The `LogisticRegression` class is a custom implementation of logistic regression
 - `summary()`: Get a summary of model parameters, standard errors, p-values, and confidence intervals.
 - `display_summary()`: Display a summary of model parameters, standard errors, p-values, and confidence intervals.
 
+### Robust Logistic Regression
+
+The `RobustLogisticRegression` class implements robust logistic regression using the Fisher scoring algorithm with epsilon-contamination for outlier resistance. This method down-weights observations that are unlikely under the main model, providing robustness against data contamination and outliers.
+
+**Parameters:**
+- `epsilon_contamination`: Contamination level (0 ≤ ε ≤ 1). Higher values provide more robustness but may reduce efficiency (default: 0.05).
+- `contamination_prob`: Probability for contamination distribution (default: 0.5).
+- `tol`: Convergence tolerance for parameter updates.
+- `max_iter`: Maximum number of iterations for the algorithm.
+- `information`: Type of information matrix to use ('expected' or 'empirical').
+- `use_bias`: Include a bias term in the model.
+- `significance`: Significance level for computing confidence intervals.
+
+**Methods:**
+- `fit(X, y)`: Fit the robust model to the data with automatic outlier down-weighting.
+- `predict(X)`: Predict target labels for input data.
+- `predict_proba(X)`: Predict class probabilities for input data.
+- `predict_ci(X)`: Predict class probabilities with confidence intervals.
+- `get_params()`: Get model parameters.
+- `set_params(**params)`: Set model parameters.
+- `summary()`: Get a summary of model parameters, standard errors, p-values, confidence intervals, and robust weights.
+- `display_summary()`: Display a comprehensive summary including robustness metrics (epsilon contamination, average/minimum robust weights).
+
+**Key Features:**
+- **Outlier Resistance**: Automatic down-weighting of observations unlikely under the main model.
+- **Robust Weights**: Access to individual observation weights showing outlier identification.
+- **Fisher Scoring Framework**: Consistent with other models using both expected and empirical information matrices.
+- **Statistical Inference**: Complete inference statistics with robust standard errors and confidence intervals.
+- **Rich Output**: Beautiful formatted summaries with robust-specific metrics and diagnostics.
+
 ### Multinomial Logistic Regression
 
 The `MultinomialLogisticRegression` class implements the Fisher Scoring algorithm for multinomial logistic regression, suitable for multi-class classification tasks.
@@ -127,34 +170,58 @@ The `FocalLossRegression` class implements the Fisher Scoring algorithm with foc
 
 ### Poisson Regression
 
-The `PoissonRegression` class implements the Fisher Scoring algorithm for Poisson regression, suitable for modeling count data.
+The `PoissonRegression` class implements the Fisher Scoring algorithm for Poisson regression, suitable for modeling count data and incidence rates. Features robust matrix operations with automatic fallback to pseudo-inverse for numerical stability.
 
 **Parameters:**
 - `max_iter`: Maximum number of iterations for optimization.
 - `epsilon`: Convergence tolerance.
 - `use_bias`: Whether to include an intercept term.
+- `offset`: Offset term for rate modeling (e.g., log exposure times).
+- `significance`: Significance level for confidence intervals.
+- `information`: Type of information matrix to use ('expected' or 'empirical').
 
 **Methods:**
 - `fit(X, y)`: Fit the model to the data.
-- `predict(X)`: Predict mean values for the Poisson model.
+- `predict(X, offset=None)`: Predict mean values with optional custom offset.
 - `calculate_st_errors(X)`: Calculate standard errors for the coefficients.
+- `summary()`: Get comprehensive model statistics including coefficients, standard errors, p-values, and confidence intervals.
+- `display_summary()`: Display beautiful formatted summary with Rich styling.
+
+**Key Features:**
+- **Offset Support**: Full support for rate modeling with log exposure times.
+- **Information Matrix Choice**: Both expected and empirical Fisher information matrices supported.
+- **Robust Implementation**: Safe matrix inversion with automatic pseudo-inverse fallback.
+- **Statistical Summaries**: Complete inference statistics with Wald tests and confidence intervals.
+- **Validated Accuracy**: Mathematical correctness verified against statsmodels with machine precision accuracy.
 
 ### Negative Binomial Regression
 
-The `NegativeBinomialRegression` class implements the Fisher Scoring algorithm for Negative Binomial regression, suitable for overdispersed count data.
+The `NegativeBinomialRegression` class implements the Fisher Scoring algorithm for Negative Binomial regression, suitable for overdispersed count data. Features enhanced robustness with comprehensive statistical inference and fixed critical implementation bugs.
 
 **Parameters:**
 - `max_iter`: Maximum number of iterations for optimization.
 - `epsilon`: Convergence tolerance.
 - `use_bias`: Whether to include an intercept term.
-- `alpha`: Fixed dispersion parameter (overdispersion adjustment for Negative Binomial).
+- `alpha`: Fixed dispersion parameter (overdispersion adjustment).
 - `phi`: Constant scale parameter.
 - `offset`: Offset term for the linear predictor.
+- `significance`: Significance level for confidence intervals.
+- `information`: Type of information matrix to use ('expected' or 'empirical').
 
 **Methods:**
 - `fit(X, y)`: Fit the model to the data.
-- `predict(X)`: Predict mean values for the Negative Binomial model.
-- `calculate_st_errors(X)`: Calculate standard errors for the coefficients.
+- `predict(X, offset=None)`: Predict mean values with proper offset handling.
+- `calculate_st_errors(X)`: Calculate standard errors with corrected implementation.
+- `summary()`: Get comprehensive model statistics including coefficients, standard errors, p-values, and confidence intervals.
+- `display_summary()`: Display beautiful formatted summary with Rich styling.
+
+**Key Improvements:**
+- **Fisher Scoring Conversion**: Converted from IWLS to proper Fisher scoring for consistency.
+- **Information Matrix Choice**: Both expected and empirical Fisher information matrices supported (empirical recommended for numerical stability).
+- **Bug Fixes**: Fixed missing offset in prediction and standard error calculations.
+- **Robust Implementation**: Safe matrix inversion with automatic pseudo-inverse fallback.
+- **Statistical Summaries**: Complete inference statistics with Wald tests and confidence intervals.
+- **Enhanced Reliability**: Comprehensive testing ensures mathematical correctness.
 
 ## Utilities
 
@@ -176,6 +243,15 @@ The package includes a utility function for visualizing observed vs predicted pr
 
 ## Change Log
 
+- **v2.0.5**
+  - **New**: Added `RobustLogisticRegression` class with epsilon-contamination for outlier-resistant classification.
+  - **Enhanced**: Poisson and Negative Binomial regression with empirical Fisher information matrix support.
+  - **Enhanced**: Converted Negative Binomial from IWLS to proper Fisher scoring for consistency.
+  - **Added**: Comprehensive offset support for Poisson regression rate modeling.
+  - **Fixed**: Critical bugs in Negative Binomial prediction and standard error calculations.
+  - **Added**: `summary()` and `display_summary()` methods with rich statistical output.
+  - **Validated**: Mathematical correctness verified against statsmodels with machine precision accuracy.
+
 - **v2.0.4**
   - Added a beta version of Poisson and Negative Binomial regression using Fisher Scoring.
   - Changed naming conventions for simplicity and consistency.

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "fisher-scoring"
-version = "2.0.4"
-description = "A Python implementation of the Fisher Scoring algorithm for classification and incidence rate tasks."
+version = "2.0.5"
+description = "A Python implementation of the Fisher Scoring algorithm for proportion and incidence rate modeling."
 authors = [
     { name = "xRiskLab", email = "contact@xrisklab.ai" }
 ]

src/fisher_scoring/__init__.py

@@ -4,6 +4,7 @@
 from .fisher_scoring_logistic import LogisticRegression
 from .fisher_scoring_multinomial import MultinomialLogisticRegression
 from .fisher_scoring_poisson import NegativeBinomialRegression, PoissonRegression
+from .fisher_scoring_robust import RobustLogisticRegression
 
 # Set up logging
 logging.basicConfig(level=logging.WARNING)
@@ -41,6 +42,7 @@ def __init__(self, *args, **kwargs):
     "FocalLossRegression",
     "PoissonRegression",
     "NegativeBinomialRegression",
+    "RobustLogisticRegression",
 ]
 
 # Add dynamic version retrieval

src/fisher_scoring/fisher_scoring_focal.py

@@ -133,7 +133,7 @@ def invert_matrix(matrix: np.ndarray) -> np.ndarray:
             print("WARNING: Singular matrix. Using pseudo-inverse.")
             return np.linalg.pinv(matrix)
 
-    def fit(self, X: np.ndarray, y: np.ndarray) -> "FisherScoringFocalRegression":
+    def fit(self, X: np.ndarray, y: np.ndarray) -> FocalLossRegression:
         """
         Fit the focal logistic regression model using Fisher scoring.
         """
@@ -168,7 +168,7 @@ def fit(self, X: np.ndarray, y: np.ndarray) -> "FisherScoringFocalRegression":
                 # Expected Fisher Information matrix
                 W_diag = (p * (1 - p) * pt).ravel()
                 information_matrix = (X.T * W_diag) @ X
-            else:
+            elif self.information == "empirical":
                 # Empirical Fisher Information matrix
                 score_vector = (y - p).reshape(X.shape[0], 1, 1)
                 X_vector = X.reshape(X.shape[0], -1, 1)
@@ -180,6 +180,11 @@ def fit(self, X: np.ndarray, y: np.ndarray) -> "FisherScoringFocalRegression":
                     * pt.reshape(-1, 1, 1),
                     axis=0,
                 )
+            else:
+                raise ValueError(
+                    f"Unknown Fisher Information type: {self.information}. Use 'expected' or 'empirical'."
+                )
+
             self.information_matrix["iteration"].append(iteration)
             self.information_matrix["information"].append(information_matrix)
 
@@ -333,7 +338,7 @@ def display_summary(self, style="default") -> None:
         summary_dict = self.summary()
 
         total_iterations = len(self.information_matrix["iteration"])
-        table = Table(title="Fisher Scoring Focal Logistic Regression Summary")
+        table = Table(title="Fisher Scoring Focal Loss Logistic Regression Summary")
 
         table.add_column(
             "Parameter",

src/fisher_scoring/fisher_scoring_logistic.py

@@ -147,7 +147,7 @@ def fit(
                 # Expected Fisher Information matrix
                 W_diag = (p * (1 - p)).ravel()
                 information_matrix = (X.T * W_diag) @ X
-            else:
+            elif self.information == "empirical":
                 # Empirical Fisher Information matrix
                 score_vector = (y - p).reshape(X.shape[0], 1, 1)
                 X_vector = X.reshape(X.shape[0], -1, 1)
@@ -158,6 +158,10 @@ def fit(
                     @ X_vector.transpose(0, 2, 1),
                     axis=0,
                 )
+            else:
+                raise ValueError(
+                    f"Unknown Fisher Information type: {self.information}. Use 'expected' or 'empirical'."
+                )
 
             self.information_matrix["iteration"].append(iteration)
             self.information_matrix["information"].append(information_matrix)

src/fisher_scoring/fisher_scoring_multinomial.py

@@ -13,9 +13,6 @@
 algorithm that is used to estimate the parameters of a multinomial logistic
 regression model.
 
-The algorithm is based on the Newton-Raphson method and uses the expected or
-empirical Fisher information matrix to update the model parameters.
-
 Additionally we provide a method to compute the standard errors, Wald statistic,
 p-values, and confidence intervals for each class.
 
@@ -142,7 +139,7 @@ def fit(
                 # Expected Fisher Information matrix
                 W_diag = (p * (1 - p)).sum(axis=1)
                 expected_I = (X.T * W_diag) @ X
-            else:
+            elif self.information == "empirical":
                 # Empirical Fisher Information matrix
                 score_vector = (y_one_hot - p).reshape(X.shape[0], -1, 1)
                 X_vector = X.reshape(X.shape[0], -1, 1)
@@ -153,6 +150,10 @@ def fit(
                     @ X_vector.transpose(0, 2, 1),
                     axis=0,
                 )
+            else:
+                raise ValueError(
+                    f"Unknown Fisher Information type: {self.information}. Use 'expected' or 'empirical'."
+                )
 
             # Select information matrix based on expected or empirical
             information_matrix = (