Skip to content

Commit bf3cf78

Browse files
Merge pull request #9 from S-FM/feat/limix
update readme
2 parents fc9aea5 + 0866b0b commit bf3cf78

File tree

1 file changed

+3
-198
lines changed

1 file changed

+3
-198
lines changed

README.md

Lines changed: 3 additions & 198 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,6 @@ Production-ready Python SDK for FAIM (Foundation AI Models) - a unified platform
1717
- **🔄 Async Support**: Built-in async/await support for concurrent requests
1818
- **📊 Rich Error Handling**: Machine-readable error codes with detailed diagnostics
1919
- **🧪 Battle-Tested**: Production-ready with comprehensive error handling
20-
- **📈 Evaluation Tools**: Built-in metrics (MSE, MASE, CRPS) and visualization utilities
21-
- **🔎 Retrieval-Augmented Inference**: Optional RAI for improved accuracy on small datasets
2220

2321
## Installation
2422

@@ -138,24 +136,6 @@ response.probabilities # Shape: (n_samples, n_classes) - classification only
138136
- **Chronos2**: ✅ Supports multivariate forecasting (multiple features)
139137
- **FlowState**: ⚠️ Univariate only - automatically transforms multivariate input
140138
- **TiRex**: ⚠️ Univariate only - automatically transforms multivariate input
141-
- **LimiX**: ✅ Supports multivariate tabular features (standard in tabular inference)
142-
143-
When you provide multivariate input (features > 1) to FlowState or TiRex, the SDK automatically:
144-
1. Issues a warning
145-
2. Forecasts each feature independently
146-
3. Reshapes the output back to your original structure
147-
148-
```python
149-
# Multivariate input to FlowState
150-
data = np.random.randn(2, 100, 3) # 2 series, 3 features
151-
request = FlowStateForecastRequest(x=data, horizon=24, prediction_type="mean")
152-
153-
# Warning: "FlowState model only supports univariate forecasting..."
154-
response = client.forecast(request)
155-
156-
# Output is automatically reshaped
157-
print(response.point.shape) # (2, 24, 3) - original structure preserved
158-
```
159139

160140
## Available Models
161141

@@ -355,177 +335,9 @@ print(response.metadata)
355335
# {'model_name': 'chronos2', 'model_version': '1.0', 'inference_time_ms': 123}
356336
```
357337

358-
## Evaluation & Metrics (Time-Series Forecasting)
359-
360-
The SDK includes a comprehensive evaluation toolkit (`faim_sdk.eval`) for measuring time-series forecast quality with standard metrics and visualizations.
361-
362-
**Note**: These metrics are designed for time-series forecasting evaluation. For tabular model evaluation (classification/regression), use standard scikit-learn metrics like `accuracy_score`, `mean_squared_error`, etc. (see tabular examples above).
363-
364-
### Installation
365-
366-
For visualization support, install with the viz extra:
367-
368-
```bash
369-
pip install faim-sdk[viz]
370-
```
371-
372-
### Available Metrics for Time-Series
373-
374-
#### Mean Squared Error (MSE)
375-
376-
Measures average squared difference between predictions and ground truth.
377-
378-
```python
379-
from faim_sdk.eval import mse
380-
381-
# Evaluate point forecast
382-
mse_score = mse(test_data, response.point, reduction='mean')
383-
print(f"MSE: {mse_score:.4f}")
384-
385-
# Per-sample MSE
386-
mse_per_sample = mse(test_data, response.point, reduction='none')
387-
print(f"MSE per sample shape: {mse_per_sample.shape}") # (batch_size,)
388-
```
389-
390-
#### Mean Absolute Scaled Error (MASE)
391-
392-
Scale-independent metric comparing forecast to naive baseline (better than MAPE for series with zeros).
393-
394-
```python
395-
from faim_sdk.eval import mase
396-
397-
# MASE requires training data for baseline
398-
mase_score = mase(test_data, response.point, train_data, reduction='mean')
399-
print(f"MASE: {mase_score:.4f}")
400-
401-
# Interpretation:
402-
# MASE < 1: Better than naive baseline
403-
# MASE = 1: Equivalent to naive baseline
404-
# MASE > 1: Worse than naive baseline
405-
```
406-
407-
#### Continuous Ranked Probability Score (CRPS)
408-
409-
Proper scoring rule for probabilistic forecasts - generalizes MAE to distributions.
410-
411-
```python
412-
from faim_sdk.eval import crps_from_quantiles
413-
414-
# Evaluate probabilistic forecast with quantiles
415-
crps_score = crps_from_quantiles(
416-
test_data,
417-
response.quantiles,
418-
quantile_levels=[0.1, 0.5, 0.9],
419-
reduction='mean'
420-
)
421-
print(f"CRPS: {crps_score:.4f}")
422-
```
423-
424-
### Visualization (Time-Series Only)
425-
426-
Plot time-series forecasts with training context and ground truth:
427-
428-
```python
429-
from faim_sdk.eval import plot_forecast
430-
431-
# Plot single sample (remember to index batch dimension!)
432-
fig, ax = plot_forecast(
433-
train_data=train_data[0], # (seq_len, features) - 2D array
434-
forecast=response.point[0], # (horizon, features) - 2D array
435-
test_data=test_data[0], # (horizon, features) - optional
436-
title="Time Series Forecast"
437-
)
438-
439-
# Save to file
440-
fig.savefig("forecast.png", dpi=300, bbox_inches="tight")
441-
```
442-
443-
#### Multi-Feature Visualization
444-
445-
```python
446-
# Option 1: All features on same plot
447-
fig, ax = plot_forecast(
448-
train_data[0],
449-
response.point[0],
450-
test_data[0],
451-
features_on_same_plot=True,
452-
feature_names=["Temperature", "Humidity", "Pressure"]
453-
)
454-
455-
# Option 2: Separate subplots per feature
456-
fig, axes = plot_forecast(
457-
train_data[0],
458-
response.point[0],
459-
test_data[0],
460-
features_on_same_plot=False,
461-
feature_names=["Temperature", "Humidity", "Pressure"]
462-
)
463-
```
464-
465-
### Complete Evaluation Example
466-
467-
```python
468-
import numpy as np
469-
from faim_sdk import ForecastClient, Chronos2ForecastRequest
470-
from faim_sdk.eval import mse, mase, crps_from_quantiles, plot_forecast
471-
472-
# Initialize client
473-
client = ForecastClient()
474-
475-
# Prepare data splits
476-
train_data = np.random.randn(32, 100, 1)
477-
test_data = np.random.randn(32, 24, 1)
478-
479-
# Generate forecast
480-
request = Chronos2ForecastRequest(
481-
x=train_data,
482-
horizon=24,
483-
output_type="quantiles",
484-
quantiles=[0.1, 0.5, 0.9]
485-
)
486-
response = client.forecast(request)
487-
488-
# Evaluate point forecast (use median)
489-
point_pred = response.quantiles[:, :, 1:2] # Extract median, keep 3D shape
490-
mse_score = mse(test_data, point_pred)
491-
mase_score = mase(test_data, point_pred, train_data)
492-
493-
# Evaluate probabilistic forecast
494-
crps_score = crps_from_quantiles(
495-
test_data,
496-
response.quantiles,
497-
quantile_levels=[0.1, 0.5, 0.9]
498-
)
499-
500-
print(f"MSE: {mse_score:.4f}")
501-
print(f"MASE: {mase_score:.4f}")
502-
print(f"CRPS: {crps_score:.4f}")
503-
504-
# Visualize best and worst predictions
505-
mse_per_sample = mse(test_data, point_pred, reduction='none')
506-
best_idx = np.argmin(mse_per_sample)
507-
worst_idx = np.argmax(mse_per_sample)
508-
509-
fig1, ax1 = plot_forecast(
510-
train_data[best_idx],
511-
point_pred[best_idx],
512-
test_data[best_idx],
513-
title=f"Best Forecast (MSE: {mse_per_sample[best_idx]:.4f})"
514-
)
515-
fig1.savefig("best_forecast.png")
516-
517-
fig2, ax2 = plot_forecast(
518-
train_data[worst_idx],
519-
point_pred[worst_idx],
520-
test_data[worst_idx],
521-
title=f"Worst Forecast (MSE: {mse_per_sample[worst_idx]:.4f})"
522-
)
523-
fig2.savefig("worst_forecast.png")
524-
```
525-
526338
## Error Handling
527339

528-
The SDK provides **machine-readable error codes** for robust error handling:
340+
The SDK provides **error codes** for robust error handling:
529341

530342
```python
531343
from faim_sdk import (
@@ -629,16 +441,9 @@ See the `examples/` directory for complete Jupyter notebook examples:
629441

630442
### Tabular Inference with LimiX
631443
- **`limix_classification_example.ipynb`** - Binary classification on breast cancer dataset
632-
- Standard approach with LimiX
633-
- Retrieval-Augmented Inference (RAI) comparison
634-
- Side-by-side metrics comparison (Accuracy, Precision, Recall, F1-Score)
635-
444+
636445
- **`limix_regression_example.ipynb`** - Regression on California housing dataset
637-
- Standard approach with LimiX
638-
- Retrieval-Augmented Inference (RAI) comparison
639-
- Comprehensive metrics comparison (MSE, RMSE, MAE, R²)
640-
- Residual statistics analysis
641-
446+
642447
## Requirements
643448

644449
- Python >= 3.10

0 commit comments

Comments
 (0)