FAI3

Welcome to your new FAI3 project and to the Internet Computer development community. By default, creating a new project adds this README and some template files to your project directory. You can edit these template files to customize your project and to include your own code to speed up the development cycle.

To get started, you might want to explore the project directory structure and the default configuration file. Working with this project in your development environment will not affect any production deployment or identity tokens.

To learn more before you start working with FAI3, see the following documentation available online:

• Quick Start
• SDK Developer Tools
• Rust Canister Development Guide
• ic-cdk
• ic-cdk-macros
• Candid Introduction

If you want to start working on your project right away, you might want to try the following commands:

cd FAI3/
dfx help
dfx canister --help

Running the project locally

If you want to test your project locally, you can use the following commands:

# Starts the replica, running in the background
dfx start --background

# Deploys your canisters to the replica and generates your candid interface
dfx deploy

Once the job completes, your application will be available at http://localhost:4943?canisterId={asset_canister_id}.

If you have made changes to your backend canister, you can generate a new candid interface with

npm run generate

at any time. This is recommended before starting the frontend development server, and will be run automatically any time you run dfx deploy.

If you are making frontend changes, you can start a development server with

npm start

Which will start a server at http://localhost:8080, proxying API requests to the replica at port 4943.

Note on frontend environment variables

If you are hosting frontend code somewhere without using DFX, you may need to make one of the following adjustments to ensure your project does not fetch the root key in production:

• setDFX_NETWORK to ic if you are using Webpack
• use your own preferred method to replace process.env.DFX_NETWORK in the autogenerated declarations
  • Setting canisters -> {asset_canister_id} -> declarations -> env_override to a string in dfx.json will replace process.env.DFX_NETWORK with the string in the autogenerated declarations
• Write your own createActor constructor

Testing

Below is a comprehensive set of testing protocols designed to validate and verify the correctness, robustness, and reliability of various evaluation metrics used in the canister. These testing protocols focus on ensuring that metrics such as Statistical Parity Difference, Disparate Impact, Average Odds Difference, Equal Opportunity Difference, Accuracy, Precision, and Recall are correctly implemented and behave as expected under a range of conditions.

Metrics Overview

• Statistical Parity Difference: Measures the difference in the probability of a positive prediction between unprivileged and privileged groups.

• Disparate Impact: Measures the ratio of positive prediction rates between unprivileged and privileged groups.

• Average Odds Difference: Evaluates fairness by comparing both False Positive Rates (FPR) and True Positive Rates (TPR) between groups.

• Equal Opportunity Difference: Focuses on the difference in True Positive Rates (TPR) between groups.

• Accuracy: Proportion of correctly classified observations (both positive and negative).

• Precision: Proportion of positively predicted observations that are actually positive.

• Recall: Proportion of actual positives that are correctly identified.

Testing Goals

• Correctness: Ensure that each metric calculation matches its mathematical definition.

• Stability: Verify metrics handle edge cases (no positives, no negatives, no data in one group, etc.) without crashing.

• Consistency: Confirm that metrics produce stable results given the same input, and that changes in input data reflect expected changes in metric values.

• Performance: Ensure that metric calculations run efficiently for large datasets without timeouts or excessive cycle consumption.

• Integration: Verify that metrics integrate well with the rest of the codebase and can be triggered via the provided APIs.

Testing Procedures

• Unit Tests for Each Metric Function:

  • Procedure: For each metric calculation function (e.g., calculate_accuracy, calculate_auc), provide a set of inputs and compare the output against a pre-computed expected result. Use small, hand-crafted datasets where the metric can be calculated by hand.

  • Pass Criteria: Outputs must match the expected value within a small tolerance for floating-point comparisons (e.g., 1e-6).

• Integration Tests with Multiple Metrics:

  • Procedure: Upload a dataset via add_dataset that contains both privileged and unprivileged groups. Call calculate_all_metrics and individually verify the returned metrics (AUC, Statistical Parity Difference, etc.). Check the metrics_history to ensure timestamps and historical metrics are recorded correctly.
  • Pass Criteria: Each metric must compute correctly, must not panic, and the metric history should contain the newly computed values with updated timestamps.

• Edge Case Testing:

  • Procedure: Upload a dataset with zero positives or zero negatives and attempt to calculate metrics that depend on these values. Upload a dataset with all data points either privileged or unprivileged and attempt fairness metric calculations. Test a scenario where predictions and labels are identical, ensuring metrics like accuracy, precision, recall, and TPR-dependent fairness measures return the correct extremes (e.g., accuracy = 1.0).
  • Pass Criteria: The canister should trap with a meaningful error message if the metric cannot be computed (no group data, no positives) or return correct boundary values (e.g., accuracy = 1.0 for a perfect classifier).

• Stress / Performance Tests:

  • Procedure: Upload a large dataset (e.g., thousands of data points) and measure the time/cycles it takes to compute metrics. Ensure no timeouts or traps occur due to performance issues.
  • Pass Criteria: Metrics are computed within a reasonable time and do not consume excessive cycles. The system remains responsive and stable under load.

• Regression Tests:

  • Procedure: Maintain a suite of known datasets and expected metrics results. After code changes or refactoring, re-run the same tests to ensure the metrics outputs remain consistent.
  • Pass Criteria: No unexpected changes in metric values for unchanged code or data, ensuring that future modifications do not break existing functionality.

Running Tests

From your project directory, run:

cargo test

Cargo will compile the code (and tests) and run all tests in your project locally.

For running integration tests, the target location of the .wasm file should be provided by an environment variable:

export TEST_WASM=/.../ICP_MVP/target/wasm32-unknown-unknown/release/FAI3_backend.wasm

The backend wasm file can be generated by this command:

cargo build --target wasm32-unknown-unknown --release -p FAI3_backend --locked

Or just use dfx build (but it takes longer).