Update main to 2025.0.2 by deronsmith · Pull Request #33 · quanted/esat

deronsmith · 2025-08-20T13:57:25Z

This pull request introduces several important updates to the ESAT project, focusing on build system improvements, documentation updates, and dependency upgrades. The most significant changes include enhancements to the build workflows for CPU and GPU wheels, updates to dependencies in the Rust backend, and refreshed documentation reflecting the new release year and improved installation instructions.

Build system and workflow improvements:

Refactored the GitHub Actions workflows in .github/workflows/python-package.yml to separately build and upload CPU and GPU wheels for all supported OSes and Python versions, improving clarity and artifact management. [1] [2]
Updated the documentation build and deployment process in .github/workflows/documentation.yml to use the latest version of the peaceiris/actions-gh-pages action and adjusted the Sphinx build and output directories for consistency.

Dependency upgrades (Rust backend):

Upgraded Rust dependencies in Cargo.toml, including numpy, nalgebra, ndarray, pyo3, indicatif, and console, and added candle-core for GPU support.

Documentation updates:

Updated documentation files to reflect the new release year (2025), improved installation instructions, added a JOSS publication badge, and clarified how to obtain development wheels. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]
Added sphinx-book-theme as a documentation requirement in doc-requirements.txt.

Other improvements:

Updated Sphinx build configuration and static assets to ensure documentation consistency and freshness. [1] [2] [3] [4]
Minor documentation corrections and clarifications, such as default parameter values and copyright year.

These changes collectively improve the maintainability, usability, and clarity of the ESAT project for both developers and users.

…ndler

…hrottle

Dev gpu

Copilot

Pull Request Overview

This pull request introduces version 2025.0.2 of the ESAT project with significant enhancements focused on GPU acceleration, improved testing infrastructure, and enhanced data handling capabilities. The most substantial changes include adding GPU support via the candle-core crate, implementing comprehensive test coverage for data imputation, and extending the data visualization and analysis capabilities.

GPU acceleration support for the Rust backend with CUDA/CANDLE integration
Comprehensive data imputation module with scikit-learn integration and test coverage
Enhanced data visualization capabilities including ridgeline plots, correlation heatmaps, and 2D histograms
Improved error handling and progress callback mechanisms

Reviewed Changes

Copilot reviewed 32 out of 54 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/data/test_imputer.py	New comprehensive test suite for data imputation functionality
rust/lib.rs	Major refactor adding GPU support, new matrix operations, and modernized API
pyproject.toml	Version bump and GPU feature flag addition
esat/data/impute.py	New data imputation module with multiple strategies
esat/data/datahandler.py	Enhanced with visualization capabilities and location-based data handling
esat/model/sa.py	Updated to support GPU acceleration and improved callback handling
esat/model/batch_sa.py	Enhanced multiprocessing with logging and progress reporting
esat/utils.py	Improved timestep calculation with better edge case handling
eval/*.py	Updated runtime evaluation scripts with GPU benchmarking
notebooks/*.ipynb	New visualization examples and updated simulation parameters

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-08-20T14:00:28Z

rust/lib.rs

+    let s_inv_data: Vec<f64> = s_data
+        .iter()
+        .map(|&x: &f64| if x > 1e-12 { 1.0 / x } else { 0.0 })
+        .collect();


Converting the tensor to F64 and then to a vector may be inefficient. Consider using the tensor's native dtype if possible, or perform the conversion only when necessary.

Suggested change

.collect();

let s_data = s.to_vec1::<f32>()?; // Extract as f32 vector

let s_data: Vec<f64> = s_data.iter().map(|&x| x as f64).collect(); // Convert to f64 if needed

let s_inv_data: Vec<f64> = s_data

.iter()

.map(|&x: &f64| if x > 1e-12 { 1.0 / x } else { 0.0 })

.collect();

Copilot · 2025-08-20T14:00:29Z

rust/lib.rs

+            &device
+        ).map_err(|e| PyErr::new::<pyo3::exceptions::PyRuntimeError, _>(format!("{}", e)))?;
+
+        // Call GPU update function (to be implemented)


The tensor creation pattern is repeated multiple times with similar parameters. Consider extracting this into a helper function to reduce code duplication and improve maintainability.

Suggested change

// Call GPU update function (to be implemented)

// Convert matrices to correct type for device

let result = if use_gpu {

// Convert to Candle Tensor using the selected device

let v_t = create_tensor_from_array(&v_arr, &device, false)?;

let u_t = create_tensor_from_array(&u_arr, &device, false)?;

let we_t = create_tensor_from_array(&we_arr, &device, false)?;

let w_t = create_tensor_from_array(&w_arr, &device, true)?;

let h_t = create_tensor_from_array(&h_arr, &device, true)?;

// Call GPU update function (to be implemented)

Copilot · 2025-08-20T14:00:29Z

rust/lib.rs

+    // Convert result to Python object
+    match result {
+        Ok((final_w, final_h, q, converged, converge_i, q_list_full)) => {
+            let result_w = final_w.t().to_pyarray(py).reshape(w.dims())?;


The variable w is used here but it's not in scope. This should likely be referencing the original w parameter from the function signature. This will cause a compilation error.

Copilot · 2025-08-20T14:00:29Z

esat/data/impute.py

+        self.missing_value = missing_value
+
+        if sklearn is None:
+            raise ImportError("scikit-learn is required for data imputation. Import esat[data] to install it.")


The error message suggests importing 'esat[data]' but according to pyproject.toml, the optional dependency is defined as 'data = ["scikit-learn"]', so the correct installation command should be 'pip install esat[data]'.

Suggested change

raise ImportError("scikit-learn is required for data imputation. Import esat[data] to install it.")

raise ImportError("scikit-learn is required for data imputation. Run 'pip install esat[data]' to install it.")

Copilot · 2025-08-20T14:00:30Z

esat/data/datahandler.py

+            logger.warning(f"Unknown file type provided. Ext: {ext}, file: {filepath}")
+            #TODO: Add custom exception for unknown file types.
+            return None
        return data


The TODO comment indicates incomplete error handling. Consider implementing a custom exception for unknown file types instead of returning None, which could lead to unexpected behavior downstream.

Copilot · 2025-08-20T14:00:30Z

esat/model/batch_sa.py

+            progress_callback=cb
+        )
        return model_i, sa



The variable self.progress_callback is accessed outside of the class context. This function appears to be a standalone function, so self is not available. This should reference the progress_callback parameter instead.

Copilot · 2025-08-20T14:00:31Z

eval/runtime_test.py

-    #         q_targets[k] = v["pmf-Q"]
+    project_dir = os.path.join("D:\\", "git", "esat")
+
+    analysis_file_path = os.path.join(project_dir, "eval", "results", "runtime_analysis.json")


The hardcoded absolute path makes this script non-portable. Consider using relative paths or environment variables to make the script work across different development environments.

Suggested change

analysis_file_path = os.path.join(project_dir, "eval", "results", "runtime_analysis.json")

# Dynamically determine the project directory as the parent of this script's directory

project_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))

analysis_file_path = os.path.join(project_dir, "eval", "results", "runtime_analysis.json")

deronsmith added 15 commits July 1, 2025 10:43

datahandler updates for plotting functions

032cbb8

cargo crate version updates

65a4828

added imputer for missing data, added location handling to the dataha…

375e471

…ndler

clean of notebooks

eaf8ed1

code cleanup process started for cargo crate updates

2801605

completed Rust crate updates, code cleanup, and GUI dataset changing

7c5f6f3

updated rust code for functional ls_nmf, includes progress_callback t…

67c810a

…hrottle

completed compiling errors update of ws-nmf refactor for GPUBackend

94d559e

rust updates completed for cpu implementation

20bd82b

bug fixes for plots

7b3bda7

model name update

12a817c

updates for analysis function returns

afb99c2

plotting output updates

a731f7b

Merge pull request #32 from quanted/dev-gpu

4d40bc7

Dev gpu

updated documentation pages

e0e98b5

Copilot AI review requested due to automatic review settings August 20, 2025 13:57

Merge branch 'main' into dev

37e1295

Copilot AI reviewed Aug 20, 2025

View reviewed changes

deronsmith merged commit 9146631 into main Aug 20, 2025
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update main to 2025.0.2#33

Update main to 2025.0.2#33
deronsmith merged 16 commits intomainfrom
dev

deronsmith commented Aug 20, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

-        // Call GPU update function (to be implemented)
+    // Convert matrices to correct type for device
+    let result = if use_gpu {
+        // Convert to Candle Tensor using the selected device
+        let v_t = create_tensor_from_array(&v_arr, &device, false)?;
+        let u_t = create_tensor_from_array(&u_arr, &device, false)?;
+        let we_t = create_tensor_from_array(&we_arr, &device, false)?;
+        let w_t = create_tensor_from_array(&w_arr, &device, true)?;
+        let h_t = create_tensor_from_array(&h_arr, &device, true)?;
+        // Call GPU update function (to be implemented)

	raise ImportError("scikit-learn is required for data imputation. Import esat[data] to install it.")
	raise ImportError("scikit-learn is required for data imputation. Run 'pip install esat[data]' to install it.")

Conversation

deronsmith commented Aug 20, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant