[Feature]: Cortex.ML Implementation

[eneshoxha]

Cortex ML SDK will be part of the Cortex Data Framework.

We are using this issue to prepare the documentation for the entire new implementation of Cortex ML, which will open new directions on data processing within Cortex Pipelines in C#.

This new library will open the doors for different real-time use cases, such as Analyzing Fraud Transactions in real time, model training, and more.

As part of this new implementation, Cortex will support different data sources e.g.,

Data Collection and Preparation

using Cortex.DataSources;

// Load transaction data
var dataSource = new DataSource("transactions.csv");
var transactions = await dataSource.LoadFromCsvAsync<Transaction>();

public class Transaction
{
    public decimal Amount { get; set; }
    public string UserId { get; set; }
    public string Location { get; set; }
    // ... other properties
}

Feature Engineering

Transform raw data into meaningful features

// Define a feature pipeline
var featurePipeline = new FeaturePipeline<Transaction>()
    .AddFeature("HourOfDay", t => t.Timestamp.Hour)
    .AddFeature("AmountDeviation", t => (t.Amount - userAvgSpending[t.UserId]) / userStdDev[t.UserId])
    .AddFeature("IsLocationUnusual", t => t.Location != userLastLocation[t.UserId]);

Model Training

Train an ML model (e.g., Random Forest) using Cortex’s built-in algorithms:

using Cortex.ML;

// Prepare labeled data (historical fraud flags)
var labeledData = transactions.Select(t => new LabeledData<Transaction>
{
    Data = t,
    Label = t.IsFraud ? 1 : 0 // 1 = Fraudulent
});

// Configure and train the model
var model = new FraudDetectionModel();
await model.TrainAsync(labeledData, featurePipeline);

// Save the model for deployment
await model.SaveAsync("fraud_model.cortex");

Using of the models

Integrate the trained model into your transaction processing flow:

// Load the pre-trained model
var model = await CortexModel.LoadAsync("fraud_model.cortex");

// For each incoming transaction, generate features and predict
public async Task<bool> IsTransactionFraudulent(Transaction transaction)
{
    var features = featurePipeline.GenerateFeatures(transaction);
    var fraudProbability = await model.PredictAsync(features);
    return fraudProbability > 0.95; // Threshold-based decision
}