@@ -17,8 +17,6 @@ Production-ready Python SDK for FAIM (Foundation AI Models) - a unified platform
1717- ** 🔄 Async Support** : Built-in async/await support for concurrent requests
1818- ** 📊 Rich Error Handling** : Machine-readable error codes with detailed diagnostics
1919- ** 🧪 Battle-Tested** : Production-ready with comprehensive error handling
20- - ** 📈 Evaluation Tools** : Built-in metrics (MSE, MASE, CRPS) and visualization utilities
21- - ** 🔎 Retrieval-Augmented Inference** : Optional RAI for improved accuracy on small datasets
2220
2321## Installation
2422
@@ -138,24 +136,6 @@ response.probabilities # Shape: (n_samples, n_classes) - classification only
138136- ** Chronos2** : ✅ Supports multivariate forecasting (multiple features)
139137- ** FlowState** : ⚠️ Univariate only - automatically transforms multivariate input
140138- ** TiRex** : ⚠️ Univariate only - automatically transforms multivariate input
141- - ** LimiX** : ✅ Supports multivariate tabular features (standard in tabular inference)
142-
143- When you provide multivariate input (features > 1) to FlowState or TiRex, the SDK automatically:
144- 1 . Issues a warning
145- 2 . Forecasts each feature independently
146- 3 . Reshapes the output back to your original structure
147-
148- ``` python
149- # Multivariate input to FlowState
150- data = np.random.randn(2 , 100 , 3 ) # 2 series, 3 features
151- request = FlowStateForecastRequest(x = data, horizon = 24 , prediction_type = " mean" )
152-
153- # Warning: "FlowState model only supports univariate forecasting..."
154- response = client.forecast(request)
155-
156- # Output is automatically reshaped
157- print (response.point.shape) # (2, 24, 3) - original structure preserved
158- ```
159139
160140## Available Models
161141
@@ -355,177 +335,9 @@ print(response.metadata)
355335# {'model_name': 'chronos2', 'model_version': '1.0', 'inference_time_ms': 123}
356336```
357337
358- ## Evaluation & Metrics (Time-Series Forecasting)
359-
360- The SDK includes a comprehensive evaluation toolkit (` faim_sdk.eval ` ) for measuring time-series forecast quality with standard metrics and visualizations.
361-
362- ** Note** : These metrics are designed for time-series forecasting evaluation. For tabular model evaluation (classification/regression), use standard scikit-learn metrics like ` accuracy_score ` , ` mean_squared_error ` , etc. (see tabular examples above).
363-
364- ### Installation
365-
366- For visualization support, install with the viz extra:
367-
368- ``` bash
369- pip install faim-sdk[viz]
370- ```
371-
372- ### Available Metrics for Time-Series
373-
374- #### Mean Squared Error (MSE)
375-
376- Measures average squared difference between predictions and ground truth.
377-
378- ``` python
379- from faim_sdk.eval import mse
380-
381- # Evaluate point forecast
382- mse_score = mse(test_data, response.point, reduction = ' mean' )
383- print (f " MSE: { mse_score:.4f } " )
384-
385- # Per-sample MSE
386- mse_per_sample = mse(test_data, response.point, reduction = ' none' )
387- print (f " MSE per sample shape: { mse_per_sample.shape} " ) # (batch_size,)
388- ```
389-
390- #### Mean Absolute Scaled Error (MASE)
391-
392- Scale-independent metric comparing forecast to naive baseline (better than MAPE for series with zeros).
393-
394- ``` python
395- from faim_sdk.eval import mase
396-
397- # MASE requires training data for baseline
398- mase_score = mase(test_data, response.point, train_data, reduction = ' mean' )
399- print (f " MASE: { mase_score:.4f } " )
400-
401- # Interpretation:
402- # MASE < 1: Better than naive baseline
403- # MASE = 1: Equivalent to naive baseline
404- # MASE > 1: Worse than naive baseline
405- ```
406-
407- #### Continuous Ranked Probability Score (CRPS)
408-
409- Proper scoring rule for probabilistic forecasts - generalizes MAE to distributions.
410-
411- ``` python
412- from faim_sdk.eval import crps_from_quantiles
413-
414- # Evaluate probabilistic forecast with quantiles
415- crps_score = crps_from_quantiles(
416- test_data,
417- response.quantiles,
418- quantile_levels = [0.1 , 0.5 , 0.9 ],
419- reduction = ' mean'
420- )
421- print (f " CRPS: { crps_score:.4f } " )
422- ```
423-
424- ### Visualization (Time-Series Only)
425-
426- Plot time-series forecasts with training context and ground truth:
427-
428- ``` python
429- from faim_sdk.eval import plot_forecast
430-
431- # Plot single sample (remember to index batch dimension!)
432- fig, ax = plot_forecast(
433- train_data = train_data[0 ], # (seq_len, features) - 2D array
434- forecast = response.point[0 ], # (horizon, features) - 2D array
435- test_data = test_data[0 ], # (horizon, features) - optional
436- title = " Time Series Forecast"
437- )
438-
439- # Save to file
440- fig.savefig(" forecast.png" , dpi = 300 , bbox_inches = " tight" )
441- ```
442-
443- #### Multi-Feature Visualization
444-
445- ``` python
446- # Option 1: All features on same plot
447- fig, ax = plot_forecast(
448- train_data[0 ],
449- response.point[0 ],
450- test_data[0 ],
451- features_on_same_plot = True ,
452- feature_names = [" Temperature" , " Humidity" , " Pressure" ]
453- )
454-
455- # Option 2: Separate subplots per feature
456- fig, axes = plot_forecast(
457- train_data[0 ],
458- response.point[0 ],
459- test_data[0 ],
460- features_on_same_plot = False ,
461- feature_names = [" Temperature" , " Humidity" , " Pressure" ]
462- )
463- ```
464-
465- ### Complete Evaluation Example
466-
467- ``` python
468- import numpy as np
469- from faim_sdk import ForecastClient, Chronos2ForecastRequest
470- from faim_sdk.eval import mse, mase, crps_from_quantiles, plot_forecast
471-
472- # Initialize client
473- client = ForecastClient()
474-
475- # Prepare data splits
476- train_data = np.random.randn(32 , 100 , 1 )
477- test_data = np.random.randn(32 , 24 , 1 )
478-
479- # Generate forecast
480- request = Chronos2ForecastRequest(
481- x = train_data,
482- horizon = 24 ,
483- output_type = " quantiles" ,
484- quantiles = [0.1 , 0.5 , 0.9 ]
485- )
486- response = client.forecast(request)
487-
488- # Evaluate point forecast (use median)
489- point_pred = response.quantiles[:, :, 1 :2 ] # Extract median, keep 3D shape
490- mse_score = mse(test_data, point_pred)
491- mase_score = mase(test_data, point_pred, train_data)
492-
493- # Evaluate probabilistic forecast
494- crps_score = crps_from_quantiles(
495- test_data,
496- response.quantiles,
497- quantile_levels = [0.1 , 0.5 , 0.9 ]
498- )
499-
500- print (f " MSE: { mse_score:.4f } " )
501- print (f " MASE: { mase_score:.4f } " )
502- print (f " CRPS: { crps_score:.4f } " )
503-
504- # Visualize best and worst predictions
505- mse_per_sample = mse(test_data, point_pred, reduction = ' none' )
506- best_idx = np.argmin(mse_per_sample)
507- worst_idx = np.argmax(mse_per_sample)
508-
509- fig1, ax1 = plot_forecast(
510- train_data[best_idx],
511- point_pred[best_idx],
512- test_data[best_idx],
513- title = f " Best Forecast (MSE: { mse_per_sample[best_idx]:.4f } ) "
514- )
515- fig1.savefig(" best_forecast.png" )
516-
517- fig2, ax2 = plot_forecast(
518- train_data[worst_idx],
519- point_pred[worst_idx],
520- test_data[worst_idx],
521- title = f " Worst Forecast (MSE: { mse_per_sample[worst_idx]:.4f } ) "
522- )
523- fig2.savefig(" worst_forecast.png" )
524- ```
525-
526338## Error Handling
527339
528- The SDK provides ** machine-readable error codes** for robust error handling:
340+ The SDK provides ** error codes** for robust error handling:
529341
530342``` python
531343from faim_sdk import (
@@ -629,16 +441,9 @@ See the `examples/` directory for complete Jupyter notebook examples:
629441
630442### Tabular Inference with LimiX
631443- ** ` limix_classification_example.ipynb ` ** - Binary classification on breast cancer dataset
632- - Standard approach with LimiX
633- - Retrieval-Augmented Inference (RAI) comparison
634- - Side-by-side metrics comparison (Accuracy, Precision, Recall, F1-Score)
635-
444+
636445- ** ` limix_regression_example.ipynb ` ** - Regression on California housing dataset
637- - Standard approach with LimiX
638- - Retrieval-Augmented Inference (RAI) comparison
639- - Comprehensive metrics comparison (MSE, RMSE, MAE, R²)
640- - Residual statistics analysis
641-
446+
642447## Requirements
643448
644449- Python >= 3.10
0 commit comments