Refactor for eif #64
Merged
Refactor for eif #64
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ing files to their place in the new structure and refactored the code so that it builds and tests pass.
… bump to v4.0 Core extraction & file renaming - Introduce IsolationTreeBase.scala and SharedTrainLogic.scala in core to unify tree building, sampling, and threshold logic. - Rename IsolationForestParams → IsolationForestParamsBase. - Move BaggedPoint.scala, Utils.scala, and model read/write code under core package. - Nodes.scala adopts new NodeBase traits for external/internal nodes. - Remove duplicate logic from IsolationForest.scala in favor of SharedTrainLogic. Build & testing updates - Introduce versions.gradle for uniform Scala & Spark version management. - Adjust test package layout for new core directory structure. Bump version to 4.0.* - The refactor potentially introduces non-backward-compatible changes, requiring a major version increment.
- Split the integration tests into two parts: - A Scala test that trains an Isolation Forest on the mammography dataset, scores the data, and exports both the model (in Avro/metadata format) and the flattened CSV scores. - A Python test that loads the exported artifacts, converts the model to ONNX, runs inference via onnxruntime, and compares the outlier scores against Spark's. - Set environment variables for Spark/Scala versions to coordinate across tests. This commit adds critical integration test coverage and ensures consistency between Spark and ONNX predictions.
…n before isolation-forest-onnx (python) tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor for Extended Isolation Forest (v4.0) – Branch Summary
Overview
This branch reorganizes our Isolation Forest codebase into a clearer, more extensible structure and bumps the version to 4.0 due to non-backward-compatible changes. It also standardizes our ScalaDoc formatting. These changes are setting the stage for a future addition of Extended Isolation Forest capability.
Key Changes
New Core Package
SharedTrainLogic
Centralizes tree-building and sampling logic that can be shared across different Isolation Forest variants.
IsolationTreeBase & NodeBase
Introduced
IsolationTreeBase.scalaand newNodeBasetraits to unify and simplify the structure of tree logic.File Moves
Moved core functionality—e.g.,
BaggedPoint.scala,Utils.scala, and model read/write code—into a dedicatedcorepackage.Refined Parameters
Param Trait Renaming
IsolationForestParamswas renamed toIsolationForestParamsBasefor a more modular design.Duplicate Logic Removal
Eliminated redundant logic in
IsolationForest.scalaby delegating toSharedTrainLogic.Build & Testing Updates
versions.gradleIntroductionCentralized Spark/Scala version management to simplify cross-version builds.
Directory Restructuring
Adjusted test package layout, placing relevant tests under the
coredirectory.Integration Tests for ONNX Conversion
isolation-forest-onnxmodule) loads those artifacts, converts the model to ONNX, and compares the ONNX inferences to Spark’s scores.Version Bump to 4.0.*
Reflects the potential for non-backward-compatible changes introduced by this refactor.
ScalaDoc Standardization
Updated all ScalaDoc comments to adhere to a consistent multi-line style, improving readability and consistency.
Validation
Comparison with Expected Values from Literature
ONNX Integration Tests
As described above, we now have ONNX integration tests that verify end-to-end compatibility. The agreement between Spark inference and inference using the same model converted to ONNX is in close agreement. On the
mammography.csvdataset, we see the following aggregating over all scored instances:Spark vs ONNX: maxDiff=0.000000238154, minDiff=0.000000000001 avgDiff=0.000000089145, medianDiff=0.000000064966.Conclusion:
The library’s internal refactor plus new ONNX integration tests provide greater clarity, stronger test coverage, and confirm strong agreement between Spark-based Isolation Forest scores and ONNX inference.