Issue: All Samples Incorrectly Classified as "No Man's Land" 🚨
- Bug:
TransformWithSafety()always returnedOutlierLevel = 4(No Man's Land) for all samples - Root Cause: When embedding statistics were uninitialized (value 0), comparison logic failed:
- All thresholds (p95, p99, extreme) were 0
- Any
min_distance > 0failed all threshold checks - Fell through to default:
outlier_level = 4(No Man's Land)
- Impact: Completely broken outlier detection - ALL samples flagged as unreliable
Fix Applied ✅:
-
Graceful Default Handling: Check if statistics are initialized before classification
- If
extreme_embedding_outlier_threshold <= 0, default toNormal(level 0) - Prevents incorrect "No Man's Land" classification
- Applied to both fast path (single-sample) and batch path
- If
-
Persistent Embedding Statistics: Statistics now saved/loaded with models
- FORMAT_VERSION bumped to 2 (backward compatible)
- Saves: mean, std, min, max, p95, p99, median, outlier thresholds
- Old model files (v1) still work - statistics recalculated on load if needed
Verification ✅:
- All 17 unit tests passing
- Save/load tests verified
- Demo runs successfully
- Backward compatible with old model files
Code Changes:
uwot_transform.cpp: Added uninitialized stats check (lines 284, 747)uwot_persistence.cpp: FORMAT_VERSION 2, save/load embedding statsuwot_simple_wrapper.h: Version string updated to "3.42.3"UMapModel.cs: Expected version updated to "3.42.3"
Issue #1: Old Models Now Work!
- FIXED: Old models (pre-v3.42.0) now rebuild
embedding_space_indexon load - Auto-calculates: Embedding statistics when loading old models
- Result: Old models work with TransformWithSafety!
- No retraining needed: Just load your old models and they work!
Issue #2: Single-Sample TransformWithSafety - CRITICAL FIX 🐛
- Bug: Calling
TransformWithSafety()with 1 sample returned all zeros - Root Cause: Fast path optimization didn't populate safety metrics
- FIXED: Enhanced fast path now supports TransformWithSafety while maintaining 12-15x speedup
- Impact: Single-sample transforms now work correctly with full safety metrics
// Before (broken): model.TransformWithSafety(oneSample); // Distance: 0, Confidence: 0 ❌ // After (fixed): model.TransformWithSafety(oneSample); // Distance: 0.001, Confidence: 0.993 ✅ // Speed: Still 12-15x faster than batch! ✅
Issue #3: Embedding Statistics - FIXED
- CRITICAL: Statistics were never calculated (all zeros in previous versions)
- FIXED: Complete statistics collection during model training
- Impact: AI safety metrics now work correctly:
- ✅ ConfidenceScore: Meaningful 0.0-1.0 range
- ✅ OutlierLevel: Proper 5-level classification (Normal → No Man's Land)
- ✅ PercentileRank: Continuous 0-100 values
- ✅ ZScore: Accurate statistical deviations
Issue #4: HNSW Ordering - FIXED
- Bug: NearestNeighborDistances[0] was farthest (confusing!)
- FIXED: Now [0] = nearest neighbor (as users expect)
v3.42.2 Production Ready:
- Cleaned debug output for production deployment
- All 17 unit tests passing
- Zero compilation warnings
- Ready for production use
After v3.42.2:
Old model loaded → embedding_space_index rebuilt → Statistics calculated
TransformWithSafety(singleSample) → Fast + Correct! ✅
Production-ready code with no debug output ✅
UMAP (Uniform Manifold Approximation and Projection) is a dimensionality reduction technique that can be used for visualization, feature extraction, and preprocessing of high-dimensional data. Unlike many other dimensionality reduction algorithms, UMAP excels at preserving both local and global structure in the data.
Example: 3D UMAP embedding rotation showing preserved data structure and clustering
For an excellent interactive explanation of UMAP, see: Understanding UMAP
This project was created specifically because existing NuGet packages and open-source C# implementations for UMAP lack critical functionality required for production machine learning applications:
- No model persistence: Cannot save trained UMAP models for reuse
- No true transform capability: Cannot project new data points using existing trained models
- No production safety features: No way to detect out-of-distribution data
- Limited dimensionality support: Restricted to 2D or 3D embeddings
- Missing distance metrics: Only basic Euclidean distance support
- No progress reporting: No feedback during long training processes
- Poor performance: Slow transform operations without optimization
- Limited production readiness: Missing essential features for real-world deployment
This implementation addresses these fundamental gaps by providing complete model persistence, authentic transform functionality, arbitrary embedding dimensions (1D-50D), multiple distance metrics, progress reporting, revolutionary HNSW optimization for 50-2000x faster training and transforms, dual HNSW architecture for AI inference, and comprehensive safety features with 5-level outlier detection - making it production-ready for AI/ML validation and real-time data quality assessment based on the proven uwot algorithm.
UMAP has several very sensetive hyper parameters namely: mind distance (dist), nr of neibours (k), localConnectivity (lC), bandwidth (usually 1 and inaccessable) and spread. The last 3 are very important if you have a large dataset and want to have a good global structure. For example look at our 3D Mamoth 1M data set that is 2d projected and subrandom sampled below:
It can be visible how k and dis is chaning the outcome a lot. (here we know the high data structure (e.g. 3d mammoth) so it is easy for us to trim these hyperparameters for a good result. see 2d projection below:
Furthermore if you increase your number of observation even if you found a good k and dist you will quickly run into problem and need to optimize IC and bandwith kernel as well.
Fine tuned parameters give pretty good result for 100k subsample
But for full 1M set data the global structure is totally off.
all of the above images are generated with our current library using the C# DEMO (data included in the project). One another this that is very important is that 99% of the global structure comes from doing an inital Spectral initialization using graph Laplacian eigenvectors an time consuming and for some dataset unstable effort. If we start from random points the outcome of UMAP is terrible specially for larger data sets. This leaves us with tuning at least four hyperparams and using a good initilization method to get good results.
my recommendation is to be very carefull with hyperparam selection or use the improved later PacMap, here is an implemetation done by me in c++ and C# PacMAP**
The Random Initialization Problem with Large Observation Data (>20k):
UMAP with random initialization cannot preserve global structure intact for large datasets regardless of hyperparameter tuning. This is a fundamental limitation:
// Random initialization - BROKEN for large data:
var largeModel = new UMapModel();
largeModel.InitMethod = InitializationMethod.Random; // Explicitly use random
var embedding = largeModel.Fit(largeDataset, embeddingDimension: 2);
// Result: Fragmented global structure, broken manifold topology
// Spectral initialization (DEFAULT in v3.40.0) - REQUIRED for quality:
var spectralModel = new UMapModel();
// InitMethod = InitializationMethod.Spectral is now the DEFAULT!
var embedding = spectralModel.Fit(largeDataset, embeddingDimension: 2);
// Result: Properly preserved global structure and manifold topologyWhy Random Initialization Fails for Large Datasets:
- Local Optima Traps: Random starting points get stuck in poor local minima
- Global Structure Loss: Cannot maintain overall dataset topology and relationships
- Fragmentation: Large datasets split into scattered clusters instead of coherent structure
- Hyperparameter Ineffective: No amount of tuning (neighbors, min_dist, etc.) can fix fundamental initialization problems
The Practical Dilemma:
- AlwaysUseSpectral = Required: Absolutely essential for quality embeddings of >20k samples
- AlwaysUseSpectral = Impractical: Extremely time-consuming (O(n²) complexity) during Fit
- Reality: 500k+ dataset with spectral initialization can take 30+ minutes vs 2-3 minutes with random
🎉 SOLVED: Verified working solution using https://github.com/libscran/umappp reference implementation!
The project now includes a real umappp integration with proper spectral initialization that solves the fragmentation issues. The mammoth dataset can now be visualized as a single, coherent structure instead of fragmented pieces.
We have successfully incorporated the libscran/umappp reference implementation, which provides:
- ✅ Spectral initialization: Non-linear manifold-aware initialization using graph Laplacian eigenvectors
- ✅ Proper epoch scheduling: Correct optimization with balanced positive/negative sampling
- ✅ Superior embeddings: Better preservation of complex 3D structures like the mammoth dataset
- ✅ HNSW-knncolle bridge: Custom interface connecting our HNSW optimization with umappp's expected neighbor search interface
- ✅ Complete C# compatibility: Maintains full API compatibility while using superior algorithms underneath
The project uses the real umappp implementation with advanced features:
├── uwot_umappp_wrapper/ # 🎯 PRIMARY: Real umappp implementation (CURRENT)
│ ├── Core umappp integration
│ │ ├── umappp.hpp # Main umappp library headers
│ │ ├── Options.hpp # Configuration and parameters
│ │ ├── Status.hpp # Status and error handling
│ │ ├── initialize.hpp # ✅ SPECTRAL INITIALIZATION (solves fragmentation!)
│ │ └── optimize_layout.hpp # Advanced optimization algorithms
│ ├── HNSW-knncolle bridge
│ │ ├── uwot_hnsw_knncolle.hpp # Custom bridge interface
│ │ └── hnswlib headers # HNSW optimization integration
│ ├── Enhanced wrapper files
│ │ ├── uwot_fit.cpp/.h # Training with umappp algorithms
│ │ ├── uwot_simple_wrapper.cpp/.h # Main API interface
│ │ ├── uwot_transform.cpp/.h # Enhanced transform with safety analysis
│ │ ├── uwot_persistence.cpp/.h # Stream-based HNSW serialization
│ │ └── All supporting modules (progress, distance, CRC32, etc.)
│ └── CMakeLists.txt # Build system with umappp dependencies
│
└── uwot_pure_cpp/ # ⚠️ LEGACY: Original implementation (DEPRECATED)
├── Reason: Fragmentation issues with complex 3D structures + compilation errors
├── Status: Abandoned due to build failures and poor performance
└── Purpose: Historical reference only - DO NOT USE
Problems with Legacy Implementation (uwot_pure_cpp):
- Build Failures: Compilation errors in test files and unstable build system
- Fragmentation Issues: Complex 3D structures like mammoth dataset fragmenting into scattered pieces
- Poor Performance: Random initialization insufficient for preserving complex manifold topology
- Outdated Architecture: Missing modern optimizations and safety features
Solution with Real umappp (uwot_umappp_wrapper):
- ✅ Spectral Initialization: Graph Laplacian eigenvectors for manifold-aware starting positions
- ✅ Proper Optimization: Correct epoch scheduling with balanced positive/negative sampling
- ✅ Better Topology Preservation: Specifically designed to handle complex non-linear structures
- ✅ Modern Feature Set: All advanced features working (AutoHNSW, dual architecture)
- ✅ Stable Builds: Reliable cross-platform compilation with comprehensive testing
To maintain our HNSW optimization benefits while using umappp, we created a custom bridge:
// Custom bridge class that implements knncolle interface for HNSW
template<typename Index_, typename Float_>
class HnswPrebuilt : public knncolle::Prebuilt<Index_, Float_, Float_> {
// Bridges our existing HNSW optimization with umappp's expected neighbor search interface
void search(Index_ i, Index_ k, std::vector<Index_>* indices,
std::vector<Float_>* distances) const override {
auto result = hnsw_index_->searchKnn(...);
}
};Benefits of This Architecture:
- ✅ Keep HNSW performance: 50-2000x faster transforms
- ✅ Get umappp quality: Superior spectral initialization and optimization
- ✅ Maintain API compatibility: Existing C# code works unchanged
- ✅ Future-proof: Easy to upgrade as umappp evolves
The production-ready umappp implementation is located in uwot_umappp_wrapper/:
# Build the real umappp implementation (RECOMMENDED)
cd uwot_umappp_wrapper
BuildWindowsOnly.bat # Windows build with Visual Studio 2022
# OR for cross-platform builds:
BuildDockerLinuxWindows.bat # Builds both Windows + Linux with Docker
# The resulting DLL has all advanced features:
# - Spectral initialization (solves fragmentation!)
# - Dual HNSW architecture (AI inference)
# - TransformWithSafety (5-level outlier detection)
# - Stream-based serialization (CRC32 validation)Note: The C# API remains identical, so existing code continues to work but now benefits from superior umappp algorithms and all advanced features.
INITIALIZATION API ENHANCEMENT: New InitializationMethod enum + spectral default + improved API clarity!
✅ InitializationMethod Enum: Clear, explicit control over initialization strategy (Auto=-1, Random=0, Spectral=1)
✅ Spectral Default: Spectral initialization now the default for best quality embeddings
✅ API Clarity: Replaced confusing AlwaysUseSpectral boolean with clearer InitMethod property
✅ Backward Compatibility: Obsolete AlwaysUseSpectral property maintained for existing code
✅ Enhanced Demo: Updated bandwidth experiments with optimal parameters (spread=2.0, local_connectivity=2.0)
✅ Dynamic Metadata: All visualizations use model-extracted parameters (no hardcoded values)
✅ Compiler Warnings Fixed: Clean compilation with zero warnings (unused variables, type conversions)
✅ Production Ready: All features tested and validated with clean builds
🔧 Critical Eigen Fix:
- Problem: Eigen library compilation failure blocking spectral initialization
- Solution: Found Eigen commit 960892ca1 (Feb 2024 JacobiSVD refactor)
- Result: ✅ Compiles successfully with MSVC, ✅ NO performance regression (2-5s tests)
Previous v3.39.0 Features: ✅ AlwaysUseSpectral Property: Force spectral initialization for any dataset size (now improved with enum) ✅ Eigen Compilation Fix: Resolved Eigen 3.4.0/3.4.1 MSVC errors and performance regression ✅ Hyperparameter Integration: LocalConnectivity & Bandwidth exposed in C# API with ModelInfo ✅ Bandwidth Sweep Testing: Comprehensive bandwidth experiments for optimal parameter discovery
Previous v3.37.0 Features: ✅ OpenMP Parallelization: 4-5x faster transforms with multi-threaded processing ✅ Single-Point Optimization: 12-15x speedup for single data point transforms (stack allocation, zero heap) ✅ Stringstream Persistence: Faster save/load with in-memory HNSW serialization (no temp files) ✅ Windows DLL Stability: Proper OpenMP cleanup prevents segfaults on DLL unload
🔥 NEW PERFORMANCE CAPABILITIES:
// Multi-point transform now 4-5x faster with OpenMP
var embeddings = model.Transform(newData); // Automatically parallelized!
// Progress: "Using 16 threads for parallel processing"
// Single-point transform now 12-15x faster
var singleEmbedding = model.Transform(singlePoint); // Fast path with stack allocationPrevious v3.34.0 Features:
- ✅ Exact k-NN Integration: Full
force_exact_knnparameter support with knncolle-based exact computation - ✅ Dual-Mode Architecture: Choose between HNSW (fast) and exact k-NN (precise) with umappp integration
- ✅ CPU Core Reporting: Real-time callback showing number of CPU cores used for parallel processing
- ✅ Parameter Propagation Fix: ALL C# parameters now properly propagate to C++ (random seed, etc.)
- ✅ Complete umappp Integration: Both HNSW and exact paths use proven libscran/umappp algorithms
- ✅ Production Ready: Extensive validation with MSE < 0.01 accuracy for both modes
🔥 NEW DUAL-MODE CAPABILITIES:
// Fast HNSW mode (default) - 50-2000x faster
var fastEmbedding = model.Fit(data, forceExactKnn: false);
// Progress: "Using 16 CPU cores for parallel processing"
// Exact k-NN mode - perfect accuracy for validation/research
var exactEmbedding = model.Fit(data, forceExactKnn: true);
// Progress: "force_exact_knn = true - using exact k-NN computation"🎯 CPU CORE MONITORING: Progress callbacks now report parallel processing capabilities:
Progress: [██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 8.0% (CPU Core Detection) - Using 16 CPU cores for parallel processing
Previous v3.16.0 Features:
- ✅ Critical Euclidean Distance Fix: L2Space squared distance properly converted with sqrt()
- ✅ Perfect Pipeline Consistency: Training embeddings match transform results exactly (MSE = 0)
- ✅ Stream-based HNSW serialization: Zero temporary files with direct memory-to-file operations
Previous v3.15.0 Features:
- ✅ Stream-Based HNSW Serialization: Direct memory-to-file operations eliminate temporary files completely
- ✅ CRC32 Data Integrity: Automatic corruption detection for both original and embedding space HNSW indices
- ✅ Dual HNSW Architecture: Original space for fitting + Embedding space for AI inference
- ✅ Memory Optimization: 80-85% reduction (40GB → ~50MB) while maintaining AI capabilities
- ✅ Speed Breakthrough: 50-2000x faster transforms with sub-millisecond AI inference
The codebase has been completely refactored into a modular architecture for maintainability, testability, and extensibility:
uwot_pure_cpp/
├── Core Engine (160 lines - 94.4% size reduction from original 2,865 lines)
│ ├── uwot_simple_wrapper.cpp/.h # Main API interface
│ └── uwot_model.cpp/.h # Model data structures
├── Specialized Modules
│ ├── uwot_fit.cpp/.h # Training algorithms
│ ├── uwot_transform.cpp/.h # Projection operations
│ ├── uwot_hnsw_utils.cpp/.h # HNSW optimization
│ ├── uwot_persistence.cpp/.h # Save/load operations
│ ├── uwot_progress_utils.cpp/.h # Progress reporting
│ └── uwot_distance.cpp/.h # Distance metrics
└── Testing & Validation
├── test_standard_comprehensive.cpp # Complete validation suite
├── test_comprehensive_pipeline.cpp # Advanced testing
└── test_error_fixes_simple.cpp # Regression tests
- 🔧 Maintainability: Individual modules can be updated independently
- 🧪 Testability: Comprehensive test suite with strict pass/fail thresholds
- 🚀 Performance: Optimized pipelines with HNSW acceleration
- 🛡️ Reliability: Modular testing prevents regressions
- 📈 Extensibility: Easy to add new distance metrics and features
The new modular architecture includes a revolutionary testing framework that catches critical bugs other tests miss:
// Comprehensive validation with strict pass/fail thresholds
test_standard_comprehensive.cpp:
├── Loss function convergence validation (ensures proper optimization)
├── Save/load projection identity testing (0.000000 MSE requirement)
├── Coordinate collapse detection (prevents normalization bugs)
├── 1% error rate validation (<0.5% threshold for HNSW approximation)
├── MSE consistency checks (fit vs transform accuracy)
└── Multi-dimensional validation (2D, 20D embeddings)Critical Bug Detection Success Story: Our comprehensive test suite caught and fixed a normalization collapse bug that standard tests completely missed. The bug caused all transform coordinates to collapse to identical values, but passed basic "function doesn't crash" tests. This demonstrates the power of result correctness validation vs. simple functional testing.
A complete, production-ready UMAP (Uniform Manifold Approximation and Projection) implementation based on the high-performance uwot R package, providing both standalone C++ libraries and cross-platform C# integration with enhanced features not available in other C# UMAP libraries.
This implementation features a revolutionary HNSW (Hierarchical Navigable Small World) optimization that replaces the traditional O(n²) brute-force k-nearest neighbor computation with an efficient O(n log n) approximate approach:
// HNSW approximate mode (default) - 50-2000x faster
var fastEmbedding = model.Fit(data, forceExactKnn: false); // Lightning fast!
// Exact mode (for validation or small datasets)
var exactEmbedding = model.Fit(data, forceExactKnn: true); // Traditional approach
// Both produce nearly identical results (MSE < 0.01)| Dataset Size | Without HNSW | With HNSW | Speedup | Memory Reduction |
|---|---|---|---|---|
| 1,000 × 100 | 2.5s | 0.8s | 3x | 75% |
| 5,000 × 200 | 45s | 1.2s | 37x | 80% |
| 20,000 × 300 | 8.5 min | 12s | 42x | 85% |
| 100,000 × 500 | 4+ hours | 180s | 80x | 87% |
- ✅ Euclidean: General-purpose data (HNSW accelerated)
- ✅ Cosine: High-dimensional sparse data (HNSW accelerated)
- ✅ Manhattan: Outlier-robust applications (HNSW accelerated)
- ⚡ Correlation: Falls back to exact computation with warnings
- ⚡ Hamming: Falls back to exact computation with warnings
The system automatically optimizes HNSW parameters and validates accuracy:
Automatic Parameter Tuning:
- Small datasets (<1,000 samples): Uses exact computation
- Large datasets (≥1,000 samples): Automatically uses HNSW with optimized parameters
- Dynamic parameter selection: Auto-tunes M, ef_construction, and ef_search based on data characteristics
- Recall validation: Validates HNSW accuracy against exact k-NN (≥95% threshold)
- ⏱️ Additional computation: Adds ~5-10% more training time for parameter optimization (well worth the accuracy gains)
Recall Validation System:
// Automatic HNSW recall validation during training
if (avg_recall >= 0.95f) {
"HNSW recall validation PASSED: 97.3% (>= 95% threshold)"
} else {
"WARNING: HNSW recall 92.1% < 95% - embeddings may be degraded. Consider increasing ef_search."
}Intelligent Fallback:
- Unsupported metrics: Automatically falls back to exact with helpful warnings
- Low recall scenarios: Automatically adjusts parameters for better accuracy
- User override: Force exact mode available for validation scenarios
| Method | Transform Speed | Memory Usage | k-NN Complexity | Accuracy Loss |
|---|---|---|---|---|
| Exact | 50-200ms | 240MB | O(n²) brute-force | 0% (perfect) |
| HNSW | <3ms | 15-45MB | O(log n) approximate | <1% (MSE < 0.01) |
Key Insight: The 50-2000x speedup comes with <1% accuracy loss, making HNSW the clear winner for production use.
// Choose your approach based on needs:
// Production applications - use HNSW (default)
var fastEmbedding = model.Fit(data, forceExactKnn: false); // 50-2000x faster!
// Research requiring perfect accuracy - use exact
var exactEmbedding = model.Fit(data, forceExactKnn: true); // Traditional approach
// Both produce visually identical embeddings (MSE < 0.01)🔥 Core Insight: Traditional UMAP implementations search neighbors in original feature space, but for AI inference, you need to search in embedding space (learned patterns) to find similar learned behaviors.
// Revolutionary two-stage HNSW indexing:
struct UwotModel {
// Stage 1: Original space HNSW - for traditional fitting/k-NN graph
std::unique_ptr<hnswlib::HierarchicalNSW<float>> original_space_index;
// Stage 2: Embedding space HNSW - for AI inference (NEW!)
std::unique_ptr<hnswlib::HierarchicalNSW<float>> embedding_space_index;
// AI safety statistics computed on embedding space
float min_embedding_distance, max_embedding_distance;
float p95_embedding_distance, p99_embedding_distance;
float mild_embedding_outlier_threshold, extreme_embedding_outlier_threshold;
};- Traditional Transform: Map new data to embedding coordinates using original space neighbors
- 🔥 REVOLUTIONARY: Search in embedding space to find similar learned patterns
// Standard transform - gets embedding coordinates
var coordinates = model.Transform(newData);
// Enhanced AI inference - finds similar learned patterns
var aiResults = model.TransformWithSafety(newData);
foreach (var result in aiResults) {
Console.WriteLine($"Similar patterns: {string.Join(", ", result.NearestNeighborIndices)}");
Console.WriteLine($"AI confidence: {result.ConfidenceScore:F3}");
Console.WriteLine($"Outlier level: {result.OutlierLevel}"); // 0=Normal, 4=NoMansLand
}Traditional Problem:
- Your AI model was trained on data X, but you don't know if new data Y is similar
- Raw feature similarity ≠ learned pattern similarity
- AI makes unreliable predictions on out-of-distribution data
Our Solution:
- Search in embedding space where patterns are learned
- Find training samples that behave similarly in the learned manifold
- Provide confidence scores and outlier detection for AI safety
var aiResults = model.TransformWithSafety(inferenceData);
foreach (var result in aiResults) {
// 5-level outlier classification in embedding space:
switch (result.OutlierLevel) {
case OutlierLevel.Normal: // AI has seen similar patterns
Console.WriteLine("✅ High confidence AI prediction");
break;
case OutlierLevel.Unusual: // Acceptable variation
Console.WriteLine("⚠️ Moderate confidence - unusual but OK");
break;
case OutlierLevel.MildOutlier: // AI extrapolating
Console.WriteLine("⚠️ Low confidence - mild outlier");
break;
case OutlierLevel.ExtremeOutlier: // AI uncertain
Console.WriteLine("❌ Very low confidence - extreme outlier");
break;
case OutlierLevel.NoMansLand: // AI should not trust
Console.WriteLine("🚨 CRITICAL: No man's land - reject prediction");
break;
}
// AI confidence score (0.0-1.0)
if (result.ConfidenceScore < 0.5) {
Console.WriteLine("AI prediction reliability questionable");
}
// Nearest training samples in embedding space
Console.WriteLine($"Most similar training patterns: {string.Join(", ", result.NearestNeighborIndices)}");
}Medical AI:
// Train on patient embeddings
var patientModel = new UMapModel();
patientModel.Fit(trainingPatientData, embeddingDimension: 15);
// In production: validate new patient data
var results = patientModel.TransformWithSafety(newPatientData);
if (results[0].OutlierLevel >= OutlierLevel.ExtremeOutlier) {
// Flag for medical review - AI prediction unreliable
AlertMedicalStaff("Patient data outside training distribution");
}Financial Trading:
// Market pattern embeddings
var marketModel = new UMapModel();
marketModel.Fit(historicalMarketData, embeddingDimension: 10);
// Live trading: detect market regime shifts
var marketResults = marketModel.TransformWithSafety(currentMarketData);
var outlierRatio = marketResults.Count(r => r.OutlierLevel >= OutlierLevel.ExtremeOutlier);
if (outlierRatio > 0.1) {
// Market conditions unlike training - pause AI trading
PauseAITrading("Market regime shift detected");
}Computer Vision Quality Control:
// Image pattern embeddings
var imageModel = new UMapModel();
imageModel.Fit(trainingImages, embeddingDimension: 20);
// Production line: detect image quality issues
var qcResults = imageModel.TransformWithSafety(productImages);
var defectCandidates = qcResults.Where(r => r.OutlierLevel >= OutlierLevel.MildOutlier);
foreach (var candidate in defectCandidates) {
FlagForHumanInspection(candidate.ImageId);
}| Feature | Traditional | Dual HNSW Architecture |
|---|---|---|
| Memory Usage | 40GB+ (store all training data) | ~50MB (store only indices) |
| AI Inference Speed | 50-200ms (linear search) | <1ms (HNSW in embedding space) |
| Pattern Similarity | Raw feature comparison | Learned pattern similarity |
| AI Confidence | Not available | 0.0-1.0 confidence scores |
| Outlier Detection | Not available | 5-level classification |
| Deployment Safety | Unknown | CRC32 validation |
Revolutionary Implementation - Zero temporary files, automatic corruption detection:
// Stream-only approach eliminates temporary files completely
void save_hnsw_to_stream_compressed(std::ostream& output, hnswlib::HierarchicalNSW<float>* hnsw_index) {
// Use stringstream to capture HNSW data for CRC computation
std::stringstream hnsw_data_stream;
hnsw_index->saveIndex(hnsw_data_stream);
// Get HNSW data as string for CRC computation
std::string hnsw_data = hnsw_data_stream.str();
uint32_t actual_size = static_cast<uint32_t>(hnsw_data.size());
// Compute CRC32 of the HNSW data
uint32_t data_crc32 = crc_utils::compute_crc32(hnsw_data.data(), actual_size);
// Write size header + CRC32 + compressed data directly to output stream
writeBinaryPOD(output, actual_size);
writeBinaryPOD(output, data_crc32);
// ... LZ4 compression and stream write
}// Automatic integrity validation for deployment safety
uint32_t original_space_crc; // Original HNSW index validation
uint32_t embedding_space_crc; // Embedding HNSW index validation
uint32_t model_version_crc; // Model structure validation
// On model load:
if (loaded_crc != saved_crc) {
// Model corruption detected - fail safely
throw new ModelCorruptionException("HNSW index corrupted - rebuild required");
}The stream-based HNSW serialization with CRC32 validation is fully implemented and tested:
[CRC32] Original space HNSW index CRC32: 5F4EE4FF
[CRC32] Embedding space HNSW index CRC32: E14C74E6
[DUAL_HNSW] Dual HNSW indices built successfully:
[DUAL_HNSW] - Original space: 150 points (4 dims) - CRC: 5F4EE4FF
[DUAL_HNSW] - Embedding space: 150 points (2 dims) - CRC: E14C74E6
[STREAM_HNSW] Stream-based serialization: SUCCESS (zero temp files)
[CRC32] Dual HNSW integrity validation: PASSED
🎯 Result: Deployment-grade reliability with automatic corruption detection, zero file management overhead, and intelligent HNSW parameter optimization with recall validation.
When you call TransformWithSafety(), you get four independent metrics that all detect outliers/OOD points, but express the result in different formats. All four metrics are calculated from the same underlying measurement: the minimum distance to the nearest neighbor in embedding space.
Why four metrics? Different use cases need different formats:
- ConfidenceScore → UI/UX (intuitive 0-100% scale)
- Severity → Business logic (categorical risk levels)
- PercentileRank → Reporting & monitoring (statistical ranking)
- Z-Score → Academic/scientific analysis (standardized statistics)
All metrics are derived from min_distance = distance from the new point to its nearest neighbor in embedding space:
| Metric | Formula | Range | Interpretation |
|---|---|---|---|
| ConfidenceScore | 1.0 - (min_distance - min_emb) / (p95_emb - min_emb) |
0.0 - 1.0 | 1.0 = Perfect match 0.7-1.0 = High confidence 0.3-0.7 = Moderate 0.0-0.3 = Low/reject |
| Z-Score | (min_distance - mean_emb) / std_emb |
-∞ to +∞ | < 2.0 = Normal 2.0-2.5 = Unusual 2.5-4.0 = Mild outlier > 4.0 = Extreme outlier |
| PercentileRank | Piecewise linear interpolation | 0 - 100 | < 80 = Well within distribution 80-95 = Approaching edge 95-99 = Rare > 99 = Extreme |
| Severity | Threshold-based classification | 5 levels | Normal (≤p95) Unusual (p95-p99) Mild (p99 to mean+2.5σ) Extreme (mean+2.5σ to mean+4σ) NoMansLand (>mean+4σ) |
Key Statistics (computed during training):
min_embedding_distance- Closest distance observed in trainingmean_embedding_distance- Average neighbor distancestd_embedding_distance- Standard deviationp95_embedding_distance- 95th percentilep99_embedding_distance- 99th percentile
Important: TransformWithSafety() always returns exactly k neighbors per point, where k = model.GetModelInfo().Neighbors (typically 15-25 depending on embedding dimension).
var results = model.TransformWithSafety(newData);
var result = results[0];
// Arrays always have length = k
Console.WriteLine(result.NearestNeighborIndices.Length); // e.g., 25
Console.WriteLine(result.NearestNeighborDistances.Length); // e.g., 25
Console.WriteLine(result.NeighborCount); // e.g., 25Edge Case - Small Datasets: If training data has fewer than k points, remaining slots are filled with sentinel values:
- Index:
0(first training point) - Distance:
1000.0(very large sentinel value)
// Detect sentinel values (if dataset was too small)
for (int i = 0; i < result.NearestNeighborDistances.Length; i++)
{
if (result.NearestNeighborDistances[i] > 100.0f) // Sentinel threshold
{
Console.WriteLine($"Only {i} real neighbors (dataset too small)");
break;
}
}using UMAPuwotSharp;
// Train model
var model = new UMapModel();
var embedding = model.Fit(trainData, embeddingDimension: 2);
// Transform new data with safety metrics
var results = model.TransformWithSafety(newData);
foreach (var result in results)
{
// Get embedding coordinates
var coords = result.ProjectionCoordinates; // float[2] for 2D
Console.WriteLine($"Projected to: [{coords[0]:F3}, {coords[1]:F3}]");
// METRIC 1: Confidence Score (0.0-1.0)
Console.WriteLine($"Confidence: {result.ConfidenceScore:F3} ({result.ConfidenceScore * 100:F1}%)");
// METRIC 2: Z-Score (standard deviations)
Console.WriteLine($"Z-Score: {result.ZScore:F2}σ");
// METRIC 3: Percentile Rank (0-100)
Console.WriteLine($"Percentile: {result.PercentileRank:F1}%");
// METRIC 4: Severity Level (categorical)
Console.WriteLine($"Severity: {result.Severity}");
// Neighbor information (always k neighbors)
Console.WriteLine($"Neighbors analyzed: {result.NearestNeighborIndices.Length}");
Console.WriteLine($"Closest training point: index={result.NearestNeighborIndices[0]}, " +
$"distance={result.NearestNeighborDistances[0]:F3}");
// Helper properties
Console.WriteLine($"Is Reliable? {result.IsReliable}");
Console.WriteLine($"Quality: {result.QualityAssessment}");
Console.WriteLine();
}Recommended Thresholds:
| Decision | Confidence | Z-Score | Percentile | Severity | Action |
|---|---|---|---|---|---|
| ✅ Auto-Accept | ≥ 0.7 | < 2.0 | < 80 | Normal | Use prediction confidently |
| 0.3-0.7 | 2.0-4.0 | 80-95 | Unusual, Mild | Flag for expert review | |
| ❌ Reject | < 0.3 | > 4.0 | > 95 | Extreme, NoMansLand | Don't trust - use fallback |
Example Decision Logic:
public class PredictionDecision
{
public enum Action { Accept, Review, Reject }
public static Action MakeDecision(TransformResult result)
{
// Strict: ALL metrics must agree for acceptance
bool highConfidence = result.ConfidenceScore >= 0.7
&& result.ZScore < 2.0
&& result.PercentileRank < 80
&& result.Severity == OutlierLevel.Normal;
if (highConfidence)
return Action.Accept;
// Reject if ANY metric shows extreme outlier
bool extremeOutlier = result.ConfidenceScore < 0.3
|| Math.Abs(result.ZScore) > 4.0
|| result.PercentileRank > 95
|| result.Severity >= OutlierLevel.Extreme;
if (extremeOutlier)
return Action.Reject;
// Middle ground: flag for review
return Action.Review;
}
}
// Usage
foreach (var result in results)
{
switch (PredictionDecision.MakeDecision(result))
{
case PredictionDecision.Action.Accept:
Console.WriteLine("✅ HIGH CONFIDENCE - Auto-accept prediction");
ProcessPrediction(result.ProjectionCoordinates);
break;
case PredictionDecision.Action.Review:
Console.WriteLine("⚠️ MODERATE - Flag for human review");
FlagForReview(result);
break;
case PredictionDecision.Action.Reject:
Console.WriteLine("❌ LOW CONFIDENCE - Reject or use fallback");
UseFallbackModel(result);
break;
}
}Example 1: High Confidence (In-Distribution)
Training statistics learned:
min_embedding_distance = 0.10
mean_embedding_distance = 0.60
std_embedding_distance = 0.20
p95_embedding_distance = 1.00
New point: min_distance = 0.40
Results:
✅ Confidence: 0.667 (67%) - Good
✅ Z-Score: -1.00σ (better than average!)
✅ Percentile: 44.3% (middle of distribution)
✅ Severity: Normal
Interpretation: "Model has seen very similar patterns. Prediction is reliable."
Action: Auto-accept
Example 2: Moderate Uncertainty (Edge of Distribution)
Training statistics learned:
min_embedding_distance = 0.10
mean_embedding_distance = 0.60
std_embedding_distance = 0.20
p95_embedding_distance = 1.00
New point: min_distance = 0.85
Results:
⚠️ Confidence: 0.167 (17%) - Low
⚠️ Z-Score: 1.25σ (slightly above average)
⚠️ Percentile: 79.1% (approaching edge)
✅ Severity: Normal (still below p95)
Interpretation: "Point is unusual but statistically acceptable. Model uncertain."
Action: Flag for human review
Example 3: Extreme Outlier (Out-of-Distribution)
Training statistics learned:
min_embedding_distance = 0.10
mean_embedding_distance = 0.60
std_embedding_distance = 0.20
p95_embedding_distance = 1.00
New point: min_distance = 1.50
Results:
❌ Confidence: 0.0 (0%) - No confidence
❌ Z-Score: 4.5σ (extreme!)
❌ Percentile: 99.0% (capped - beyond distribution)
❌ Severity: NoMansLand
Interpretation: "Model has NEVER seen anything like this. Pure extrapolation."
Action: Reject prediction - use fallback or request human input
1. Industry-Specific Thresholds:
| Industry | Accept Threshold | Review Threshold | Why |
|---|---|---|---|
| Medical AI | Confidence ≥ 0.9, Severity = Normal | 0.7-0.9 | Lives at stake - be conservative |
| Financial Trading | Confidence ≥ 0.8, Z-Score < 1.5 | 0.5-0.8 | Money at risk - cautious but actionable |
| E-Commerce Recommendations | Confidence ≥ 0.5, Percentile < 90 | 0.3-0.5 | Low stakes - more tolerance |
| Fraud Detection | Percentile > 95, Severity ≥ Mild | 80-95 | Catch outliers - opposite logic |
2. Monitoring & Alerting:
// Track OOD rate over time
var oodCount = results.Count(r => r.Severity >= OutlierLevel.Mild);
var oodRate = (double)oodCount / results.Length;
if (oodRate > 0.15) // More than 15% outliers
{
Console.WriteLine($"⚠️ WARNING: High OOD rate ({oodRate:P0}) - model may need retraining");
AlertDataScientists("OOD rate exceeds threshold", oodRate);
}3. Batch Processing with Filtering:
// Process batch and separate by confidence
var reliableResults = results.Where(r => r.IsReliable).ToList();
var uncertainResults = results.Where(r => !r.IsReliable).ToList();
Console.WriteLine($"✅ Reliable predictions: {reliableResults.Count} ({reliableResults.Count * 100.0 / results.Length:F1}%)");
Console.WriteLine($"⚠️ Uncertain predictions: {uncertainResults.Count} ({uncertainResults.Count * 100.0 / results.Length:F1}%)");
// Process reliable predictions automatically
ProcessAutomatically(reliableResults);
// Queue uncertain predictions for human review
QueueForReview(uncertainResults);4. Dataset Size Validation:
// CRITICAL: Ensure training data is larger than k
public float[,] Fit(float[,] data, int? nNeighbors = null, ...)
{
int nSamples = data.GetLength(0);
int actualNeighbors = nNeighbors ?? CalculateOptimalNeighbors(embeddingDimension);
if (actualNeighbors >= nSamples)
{
throw new ArgumentException(
$"Number of neighbors ({actualNeighbors}) must be less than dataset size ({nSamples}). " +
$"Either reduce nNeighbors or add more training data.");
}
return CallFit(data, actualNeighbors, ...);
}5. Logging for Debugging:
// Log detailed metrics for debugging production issues
var logger = LogManager.GetCurrentClassLogger();
foreach (var result in results)
{
logger.Debug($"OOD Metrics - " +
$"Confidence: {result.ConfidenceScore:F3}, " +
$"Z-Score: {result.ZScore:F2}, " +
$"Percentile: {result.PercentileRank:F1}%, " +
$"Severity: {result.Severity}, " +
$"Neighbors: {result.NearestNeighborIndices.Length}, " +
$"MinDist: {result.NearestNeighborDistances.Min():F3}");
}- All four metrics detect OOD - they just express it in different formats for different audiences
- Always get k neighbors - arrays always have length =
model.Neighbors(with sentinels if dataset too small) - Use multiple metrics together - more robust than relying on one alone
- Calibrate thresholds - adjust based on your industry's risk tolerance
- Monitor OOD rates - high OOD rate indicates need for model retraining
- Validate dataset size - ensure training data has more samples than k
For more details on the dual HNSW architecture powering these metrics, see the section above.
Explicit, type-safe control over UMAP initialization strategy with spectral as default!
using UMAPuwotSharp;
// NEW v3.40.0: InitializationMethod enum for clear API
var model = new UMapModel();
// Spectral initialization (DEFAULT - best quality)
model.InitMethod = InitializationMethod.Spectral; // Explicit control
// Or use Auto mode (size-based selection)
model.InitMethod = InitializationMethod.Auto; // ≤20k: Spectral, >20k: Random
// Or force Random (fast but lower quality)
model.InitMethod = InitializationMethod.Random; // Fast initialization
var embedding = model.FitWithProgress(
data: complex3DData,
progressCallback: progress => Console.WriteLine($"Init: {progress.Message}"),
embeddingDimension: 2,
nNeighbors: 80,
minDist: 0.35f,
localConnectivity: 1.3f, // Fuzzy simplicial set parameter
bandwidth: 3.2f // Kernel density estimation bandwidth
);
// Benefits:
// ✅ Clear, explicit API with enum instead of confusing boolean
// ✅ Spectral DEFAULT: Best quality embeddings out-of-the-box
// ✅ Manifold-aware starting positions using graph Laplacian eigenvectors
// ✅ Better preservation of complex 3D structures (solves mammoth fragmentation!)
// ✅ Superior topology preservation for non-linear manifolds
// ⚠️ Slower initialization (spectral O(n²) vs random O(n)) but worth the quality gainInitializationMethod Options:
- Spectral (1): High-quality manifold-aware initialization (DEFAULT)
- Auto (-1): Automatic selection based on dataset size (≤20k: Spectral, >20k: Random)
- Random (0): Fast random initialization (lower quality for large datasets)
Backward Compatibility:
// OLD API still works (marked obsolete)
model.AlwaysUseSpectral = true; // Equivalent to: model.InitMethod = InitializationMethod.Spectral
model.AlwaysUseSpectral = false; // Equivalent to: model.InitMethod = InitializationMethod.AutoProduction Strategy:
// v3.40.0: Spectral is now the DEFAULT - no need to set explicitly!
var model = new UMapModel();
// InitMethod = InitializationMethod.Spectral (automatic)
var embedding = model.Fit(largeDataset); // Best quality by default!
// For speed-critical applications, you can opt for Auto mode:
var fastModel = new UMapModel();
fastModel.InitMethod = InitializationMethod.Auto; // Size-based selection
var embedding = fastModel.Fit(largeDataset);
// Alternative: Use smaller representative subset for spectral init,
// then apply learned parameters to full datasetPerfect for:
- Complex 3D structures like mammoth, helix, or manifold datasets
- High-quality visualizations where embedding quality matters more than speed
- Research applications where optimal manifold preservation is critical
- Large datasets (>20k): Essential requirement despite time cost
- Datasets that fragment with random initialization
- 10k samples: ~30-60 seconds with spectral init
- 50k samples: ~5-10 minutes with spectral init
- 100k samples: ~20-40 minutes with spectral init
- Random init: 10-100x faster but produces broken embeddings for large datasets
This is NOT an implementation issue - it's an inherent limitation of the UMAP algorithm itself.
UMAP's optimization process suffers from a fundamental algorithmic weakness when dealing with large datasets (>20k samples) using random initialization:
// This problem exists in ALL UMAP implementations (Python, R, C++, etc.)
var largeDataset = LoadLargeData(); // 50k+ samples
var model = new UMapModel();
var embedding = model.Fit(largeDataset); // Random init for large datasets by default
// Result: Fragmented, broken global structure regardless of:
// - Hyperparameter tuning (n_neighbors, min_dist, spread)
// - Implementation quality (Python, R, C#, etc.)
// - Hardware performance (CPU, GPU, memory)
// - Algorithm variants (umappp, uwot, original UMAP)The problem stems from UMAP's gradient descent optimization on high-dimensional manifolds:
- Local Optima Traps: Random starting points in high-dimensional space get trapped in poor local minima
- Manifold Curvature: Complex global structures are difficult to preserve without manifold-aware initialization
- Optimization Landscape: Large datasets create extremely complex loss surfaces that random initialization cannot navigate effectively
- Fundamental Mathematics: The fuzzy simplicial set construction requires good initial positions to maintain global topology
This affects ALL UMAP implementations:
- ✅ Python UMAP: Same random initialization problems for >20k samples
- ✅ R uwot: Identical algorithmic limitations
- ✅ Our C++ implementation: Same mathematical constraints as reference implementations
- ❌ Cannot be fixed: No amount of implementation optimization can overcome this fundamental limitation
For large datasets requiring dimensionality reduction, consider algorithms specifically designed to handle random initialization effectively:
PacMap (Pairwise Controlled Manifold Approximation) is specifically designed to overcome UMAP's limitations:
// PacMap: Better choice for large datasets with random initialization
// Available at: https://github.com/78Spinoza/PacMapDotnet
using PacMap;
var pacmap = new PacMapModel();
var embedding = pacmap.Fit(largeDataset, embeddingDimension: 2);
// Advantages:
// ✅ Works excellently with random initialization even for 100k+ samples
// ✅ Preserves global structure without spectral initialization overhead
// ✅ Fast processing (minutes, not hours) for large datasets
// ✅ Designed specifically to overcome UMAP's large dataset limitations
// ✅ Better preservation of both local and global structureWhy PacMap Works Better for Large Datasets:
- Anchored Optimization: Uses carefully selected anchor points to guide embedding
- Pairwise Preservation: Directly optimizes pairwise relationships rather than fuzzy sets
- Random-Init Robust: Algorithm design inherently works well with random starting positions
- Scalable Architecture: Specifically optimized for large dataset performance
| Dataset Size | Recommended Algorithm | Reason |
|---|---|---|
| < 20k samples | UMAP (auto-spectral) | Best quality, reasonable speed |
| 20k-50k samples | UMAP (force spectral) | Accept quality, plan processing time |
| > 50k samples | PacMap | Superior large dataset performance |
| Production systems | PacMap | Reliable fast processing without spectral overhead |
| Research quality | UMAP (spectral) | Best possible embedding quality |
Our implementation provides the best possible UMAP experience:
- ✅ Spectral initialization: The only solution for UMAP large dataset quality
- ✅ Hyperparameter control: Bandwidth & LocalConnectivity for fine-tuning
- ✅ Performance optimization: HNSW acceleration for all other operations
- ✅ Quality validation: Comprehensive testing and verification
But we cannot overcome UMAP's fundamental algorithmic limitations. For large datasets requiring both quality AND speed, PacMap is the mathematically superior choice.
- Small datasets (<20k): UMAP with auto-spectral initialization is excellent
- Medium datasets (20k-50k): UMAP with forced spectral initialization (accept time cost)
- Large datasets (>50k): PacMap is the recommended solution for production use
- Research quality: UMAP with spectral initialization (budget processing time)
This recommendation is based on mathematical algorithmic properties, not implementation preferences.
This may be a fundamental issue with how UMAP is typically used for large datasets!
The Hidden Hyperparameter Problem
Standard UMAP Documentation (Incomplete):
- Main hyperparameters: n_neighbors, min_dist, spread
- Bandwidth: Fixed at 1.0 (never mentioned or exposed)
The Reality We Discovered:
- For large datasets (50k-100k+) with random initialization, bandwidth=1.0 creates too sparse a graph
- You need bandwidth=2.0-3.0 to get proper global structure
- Most UMAP implementations hardcode bandwidth=1.0 and don't expose it!
// STANDARD UMAP (with hidden bandwidth=1.0)
var standardModel = new UMapModel();
var embedding = standardModel.Fit(largeDataset, embeddingDimension: 2);
// bandwidth = 1.0 (hardcoded, invisible)
// Result: Fragmented global structure for large datasets
// ENHANCED UMAP (with exposed bandwidth)
var enhancedModel = new UMapModel();
enhancedModel.AlwaysUseSpectral = false; // Random init
var embedding = enhancedModel.FitWithProgress(
data: largeDataset,
embeddingDimension: 2,
bandwidth: 3.0f // CRITICAL: Exposed bandwidth for large datasets!
);
// Result: Proper global structure preserved even with random initThe Complete UMAP Hyperparameters Should Be:
- n_neighbors - Local neighborhood size (15-80)
- min_dist - Minimum point separation in embedding (0.0-0.99)
- spread - Global scale factor (0.5-5.0)
- bandwidth - Fuzzy kernel width (CRITICAL for large datasets!) (1.0-5.0)
The Bandwidth Effect on Large Datasets:
| Dataset Size | Standard bandwidth=1.0 | Enhanced bandwidth=2.5-3.0 |
|---|---|---|
| < 10k | Works fine | Slightly better |
| 10k-20k | Minor fragmentation | Good structure |
| 20k-50k | Noticeable fragmentation | Much better global structure |
| 50k-100k | Severe fragmentation | Proper global preservation |
| > 100k | Broken global structure | Usable large dataset embeddings |
The Root Cause:
- Python's umap-learn: Bandwidth hardcoded to 1.0, not exposed
- R's uwot: Bandwidth hardcoded to 1.0, not exposed
- Most implementations: Copy the same limitation
- Documentation: Never mentions bandwidth as a tunable parameter
User Experience:
# Python UMAP - users frustrated with large datasets
import umap
reducer = umap.UMAP(n_neighbors=15, min_dist=0.1)
embedding = reducer.fit_transform(large_data) # bandwidth=1.0 hidden
# Result: Fragmented embeddings, users think UMAP doesn't work for large dataOur Enhanced Solution:
// C# UMAP with exposed bandwidth - users can succeed with large data
var model = new UMapModel();
var embedding = model.FitWithProgress(
data: largeData,
nNeighbors: 15,
minDist: 0.1f,
bandwidth: 3.0f // EXPOSED: Critical for large dataset success!
);
// Result: Proper global structure, users succeed with large datasetsUMAP's Fuzzy Simplicial Set Construction:
// Standard UMAP bandwidth=1.0:
float val = (distance - rho[i]) / 1.0; // Fixed bandwidth
weights[k] = (val <= 0) ? 1.0 : std::exp(-val); // Too sparse for large datasets
// Enhanced UMAP bandwidth=3.0:
float val = (distance - rho[i]) / 3.0; // Larger bandwidth
weights[k] = (val <= 0) ? 1.0 : std::exp(-val); // Proper connectivity for large datasetsBandwidth controls the fuzzy kernel width:
- Low bandwidth (1.0): Very tight fuzzy connections → sparse graph → fragmentation
- High bandwidth (2.5-3.0): Wider fuzzy connections → denser graph → global coherence
Before This Discovery:
- Large datasets (>20k) with random init → impossible to get good results
- Users forced to use spectral init (30+ minutes) or give up
- UMAP considered "not suitable for large datasets"
After This Discovery:
- Large datasets with random init + proper bandwidth → excellent results!
- Processing time: 2-5 minutes instead of 30+ minutes
- UMAP becomes practical for production large dataset use
The Implications:
- UMAP is fundamentally better for large datasets than anyone thought
- The "spectral init required" narrative may be overstated
- Most UMAP implementations are handicapping users by hiding bandwidth
- Large dataset UMAP could be revolutionized by exposing this parameter
// Bandwidth recommendations for random initialization:
float GetOptimalBandwidth(int datasetSize) {
if (datasetSize < 10000) return 1.0f; // Standard works fine
if (datasetSize < 25000) return 1.5f; // Slightly wider kernel
if (datasetSize < 50000) return 2.0f; // Moderate expansion
if (datasetSize < 100000) return 2.5f; // Significant expansion
return 3.0f; // Maximum for very large datasets
}Our bandwidth sweep experiments demonstrate this:
DemoHairyMammothBandwidthExperiments()tests bandwidth [1.0-2.8]- Results show dramatic improvement in global structure preservation
- Large datasets become usable with proper bandwidth tuning
This could be one of the most important UMAP discoveries in recent years!
UMAP's Heavy Reliance on Initialization is "Like Cheating"
The fact that UMAP requires spectral or PCA initialization to maintain global structure reveals a fundamental weakness in the algorithm:
// This is essentially "cheating" - UMAP shouldn't need perfect initialization:
var spectralModel = new UMapModel();
spectralModel.AlwaysUseSpectral = true; // Gives UMAP the "answer" upfront
var embedding = spectralModel.Fit(largeDataset);
// Result: Good global structure, but UMAP didn't discover it organically
// Real test: Can UMAP find global structure from random initialization?
var randomModel = new UMapModel();
randomModel.AlwaysUseSpectral = false; // Honest test
var embedding = randomModel.FitWithProgress(
data: largeDataset,
bandwidth: 3.0f // Even with optimal bandwidth...
);
// Result: Still incomplete global structure preservationThe Fundamental Issue:
- Good algorithms should work well with random initialization
- UMAP relies heavily on spectral/PCA initialization to "give it the answer"
- This is algorithmic crutch - not a strength of the core optimization
Even with Optimal Bandwidth, Limitations Remain:
Our testing shows that even with bandwidth=2.5-3.0:
- ✅ Significant improvement over bandwidth=1.0
- ✅ Better local structure preservation
- ✅ Reduced fragmentation compared to standard UMAP
- ❌ Still not perfect global structure like spectral initialization
- ❌ Some topological distortions remain in complex manifolds
What This Tells Us About UMAP:
- Algorithm Dependency: UMAP's optimization gets trapped in local minima without good initialization
- Optimization Weakness: The gradient descent approach cannot reliably discover global structure
- Hidden Requirements: "Production UMAP" secretly requires initialization preprocessing
- Algorithm Limitation: Even with all hyperparameters exposed, random init has fundamental limits
The Implications for Users:
// Reality check for UMAP users:
var model = new UMapModel();
model.AlwaysUseSpectral = true; // Most users need this for quality
var embedding = model.Fit(largeDataset);
// Truth: You're not just using UMAP - you're using UMAP + spectral initialization
// The spectral init is doing most of the heavy lifting for global structureBetter Alternatives for True Large Dataset Performance:
If an algorithm requires extensive initialization preprocessing to work well:
// Consider algorithms designed to work well with random initialization:
// 1. PacMap - Specifically designed for random init robustness
// 2. t-SNE - Works reasonably well with random init (though slow)
// 3. Force-directed methods - Don't rely on initialization quality
// 4. Modern embeddings that don't need "cheating" with initializationBottom Line on UMAP + Large Datasets:
- Bandwidth tuning: Major improvement over standard implementations
- Still limited: Cannot fully overcome initialization dependency
- Spectral required: For truly high-quality large dataset embeddings
- "Cheating" necessary: UMAP needs initialization help to maintain global structure
- Alternative consideration: For production systems, algorithms that work well organically may be better
This is an honest assessment: UMAP is a powerful algorithm but has fundamental limitations for large datasets that even proper hyperparameter tuning cannot completely overcome.
Comprehensive bandwidth experiments with LocalConnectivity integration!
// NEW: Bandwidth sweep testing for optimal kernel density estimation
var model = new UMapModel();
model.AlwaysUseSpectral = true; // Pair with spectral for best quality
// Bandwidth sweep experiments automatically test values [1.0, 1.2, 1.4, 1.6, 1.8, 2.0, 2.2, 2.4, 2.8]
// DemoHairyMammothBandwidthExperiments() in UMAPDemo/Program.cs:
// - Tests 9 bandwidth values with fixed local_connectivity=1.3
// - Generates visualizations with hyperparameters in titles
// - Provides quality analysis and best bandwidth recommendation
// - Saves to Results/hairy_mammoth_bandwidth_experiments/
// Manual bandwidth testing:
var embedding = model.FitWithProgress(
data: complexData,
embeddingDimension: 2,
nNeighbors: 45,
minDist: 0.35f,
localConnectivity: 1.3f, // Fixed optimal local connectivity
bandwidth: 2.4f, // Test different bandwidth values
AlwaysUseSpectral: true // Enable spectral for best quality
);
// NEW: Access hyperparameters from ModelInfo
var info = model.ModelInfo;
Console.WriteLine($"Local Connectivity: {info.LocalConnectivity}"); // 1.3
Console.WriteLine($"Bandwidth: {info.Bandwidth}"); // 2.4
Console.WriteLine($"Neighbors: {info.Neighbors}"); // 45
// Enhanced visualization titles now include all hyperparameters:
// "Mammoth 100k Points - UMAP 2D Embedding"
// "UMAP v3.39.0 | Sample: 100,000 | HNSW"
// "k=80 | Euclidean | dims=2"
// "min_dist=0.35 | spread=1.0"
// "local_connectivity=1.3 | bandwidth=3.2" // NEW: Enhanced titles
// "HNSW: M=48, ef_c=600, ef_s=200"Bandwidth Guidelines:
- Low bandwidth (1.0-1.4): Tight local structure, more clustering
- Medium bandwidth (1.6-2.2): Balanced local/global structure
- High bandwidth (2.4-2.8): Looser local structure, better global preservation
LocalConnectivity & Bandwidth Interaction:
- local_connectivity=1.3: Standard fuzzy simplicial set construction
- bandwidth=3.2: Enhanced kernel density estimation for hairy mammoth 100k
- Combination: Optimal for complex 3D structures with spectral initialization
Complete spread parameter implementation with dimension-aware defaults!
// Automatic spread optimization based on dimensions
var embedding2D = model.Fit(data, embeddingDimension: 2); // Auto: spread=5.0 (t-SNE-like)
var embedding10D = model.Fit(data, embeddingDimension: 10); // Auto: spread=2.0 (balanced)
var embedding27D = model.Fit(data, embeddingDimension: 27); // Auto: spread=1.0 (compact)
// Manual spread control for fine-tuning
var customEmbedding = model.Fit(data,
embeddingDimension: 2,
spread: 5.0f, // Space-filling visualization
minDist: 0.35f, // Minimum point separation
nNeighbors: 25 // Optimal for 2D visualization
);
// Research-backed optimal combinations:
// 2D Visualization: spread=5.0, minDist=0.35, neighbors=25
// 10-20D Clustering: spread=1.5-2.0, minDist=0.1-0.2
// 24D+ ML Pipeline: spread=1.0, minDist=0.1- HNSW optimization: 50-2000x faster with 80-85% memory reduction
- Arbitrary dimensions: 1D to 50D embeddings with memory estimation
- Multiple distance metrics: Euclidean, Cosine, Manhattan, Correlation, Hamming
- Smart spread defaults: Automatic optimization based on embedding dimensions
- Real-time progress reporting: Phase-aware callbacks with time estimates
- Model persistence: Save/load trained models efficiently
- Safety features: 5-level outlier detection for AI validation
using UMAPuwotSharp;
// Create model with enhanced features
using var model = new UMapModel();
// NEW v3.40.0: Spectral initialization is now the DEFAULT!
// No need to set InitMethod unless you want Auto or Random
// model.InitMethod = InitializationMethod.Spectral; // This is the default
// Train with all features: Spectral + HNSW + smart defaults + progress reporting
var embedding = model.FitWithProgress(
data: trainingData,
progressCallback: progress => Console.WriteLine($"Training: {progress.PercentComplete:F1}%"),
embeddingDimension: 20, // Higher dimensions for ML pipelines
spread: 2.0f, // Balanced manifold preservation
minDist: 0.1f, // Optimal for clustering
nNeighbors: 30, // Good for 20D
nEpochs: 300,
metric: DistanceMetric.Cosine, // HNSW-accelerated!
forceExactKnn: false, // Use HNSW optimization (50-2000x faster)
localConnectivity: 1.3f, // Local connectivity for fuzzy simplicial set
bandwidth: 3.2f // Bandwidth for kernel density estimation
);
// Save model with enhanced metadata
model.SaveModel("production_model.umap");
// Load and use model
using var loadedModel = UMapModel.LoadModel("production_model.umap");
// NEW: Access hyperparameters from ModelInfo
Console.WriteLine($"Initialization: {loadedModel.InitMethod}"); // Spectral
Console.WriteLine($"Local Connectivity: {loadedModel.ModelInfo.LocalConnectivity}");
Console.WriteLine($"Bandwidth: {loadedModel.ModelInfo.Bandwidth}");
// Transform with safety analysis
var results = loadedModel.TransformWithSafety(newData);
foreach (var result in results)
{
if (result.OutlierSeverity >= OutlierLevel.MildOutlier)
{
Console.WriteLine($"Warning: Outlier detected (confidence: {result.ConfidenceScore:F3})");
}
}v3.13.0 Enhanced Binaries:
- Windows x64:
uwot.dll- Complete HNSW + spread parameter implementation - Linux x64:
libuwot.so- Full feature parity with spread optimization
Features: Multi-dimensional support, smart spread defaults, HNSW optimization, progress reporting, and cross-platform compatibility. Ready for immediate deployment.
- Preserves local structure: Keeps similar points close together
- Maintains global structure: Preserves overall data topology effectively
- Scalable: Handles large datasets efficiently
- Fast: High-performance implementation optimized for speed
- Versatile: Works well for visualization, clustering, and as preprocessing
- Deterministic: Consistent results across runs (with fixed random seed)
- Flexible: Supports various distance metrics and custom parameters
- Multi-dimensional: Supports any embedding dimension from 1D to 50D
- Production-ready: Comprehensive safety features for real-world deployment
- Parameter sensitivity: Results can vary significantly with parameter changes
- Interpretation challenges: Distances in embedding space don't always correspond to original space
- Memory usage: Can be memory-intensive for very large datasets (e.g., 100k samples × 300 features typically requires ~4-8GB RAM during processing, depending on n_neighbors parameter)
- Mathematical complexity: The underlying theory is more complex than simpler methods like PCA
Currently available UMAP libraries for C# (including popular NuGet packages) have significant limitations:
- No model persistence: Cannot save trained models for later use
- No true transform capability: Cannot embed new data points using pre-trained models
- Limited dimensionality: Usually restricted to 2D or 3D embeddings only
- Single distance metric: Only Euclidean distance supported
- No progress feedback: No way to monitor training progress
- Performance issues: Often slower implementations without the optimizations of uwot
- Limited parameter support: Missing important UMAP parameters and customization options
This enhanced implementation addresses ALL these gaps by providing:
- True model persistence: Save and load trained UMAP models in efficient binary format
- Authentic transform functionality: Embed new data using existing models (essential for production ML pipelines)
- Smart spread parameter (NEW v3.1.2): Dimension-aware defaults for optimal embeddings
- Arbitrary dimensions: Support for 1D to 50D embeddings including specialized dimensions like 27D
- Multiple distance metrics: Five different metrics optimized for different data types
- HNSW optimization: 50-2000x faster with 80-85% memory reduction
- Real-time progress reporting: Live feedback during training with customizable callbacks
- Complete parameter support: Full access to UMAP's hyperparameters including spread
// Train UMAP on your AI training dataset
var trainData = LoadAITrainingData();
using var umapModel = new UMapModel();
var embeddings = umapModel.Fit(trainData, embeddingDimension: 10);
// Train your AI model using UMAP embeddings (often improves performance)
var aiModel = TrainAIModel(embeddings, labels);
// In production: Validate new inference data
var results = umapModel.TransformWithSafety(newInferenceData);
foreach (var result in results) {
if (result.Severity >= OutlierLevel.Extreme) {
LogUnusualInput(result); // Flag for human review
}
}Monitor if your production data drifts from training distribution:
var productionBatches = GetProductionDataBatches();
foreach (var batch in productionBatches) {
var results = umapModel.TransformWithSafety(batch);
var outlierRatio = results.Count(r => r.Severity >= OutlierLevel.Extreme) / (float)results.Length;
if (outlierRatio > 0.1f) { // More than 10% extreme outliers
Console.WriteLine($"⚠️ Potential data drift detected! Outlier ratio: {outlierRatio:P1}");
Console.WriteLine($" Consider retraining your AI model.");
}
}// Feature extraction for downstream ML models
var features27D = model.Fit(highDimData, embeddingDimension: 27, metric: DistanceMetric.Cosine);
// Use as input to neural networks, clustering algorithms, etc.// Compare different distance metrics for the same data
var metrics = new[] {
DistanceMetric.Euclidean,
DistanceMetric.Cosine,
DistanceMetric.Manhattan
};
foreach (var metric in metrics)
{
var embedding = model.Fit(data, metric: metric, embeddingDimension: 2);
// Analyze which metric produces the best clustering/visualization
}// Long-running training with progress tracking
var embedding = model.FitWithProgress(
largeDataset,
progressCallback: (epoch, total, percent) =>
{
// Log to monitoring system
logger.LogInformation($"UMAP Training: {percent:F1}% complete");
// Update database/UI
await UpdateTrainingProgress(percent);
},
embeddingDimension: 10,
nEpochs: 1000,
metric: DistanceMetric.Correlation
);Production-ready umappp implementation with all advanced features:
- ✅ Real umappp Integration: Complete libscran/umappp reference implementation
- ✅ Spectral Initialization: Manifold-aware graph Laplacian eigenvector initialization
- ✅ AutoHNSW Optimization: 50-2000x faster with automatic parameter tuning
- ✅ Dual HNSW Architecture: Original + embedding space indices for AI inference
- ✅ TransformWithSafety: 5-level outlier detection with confidence scoring
- ✅ Stream-based Serialization: CRC32 validation, zero temporary files
- ✅ Fragmentation Solution: Specifically handles complex 3D structures like mammoth dataset
- Complete Advanced Feature Set: All modern optimizations working correctly
- Multiple Distance Metrics: Euclidean, Cosine, Manhattan, Correlation, Hamming
- Arbitrary Dimensions: Support for 1D to 50D embeddings
- Enhanced Progress Reporting: Real-time training feedback with phase information
- Production Model Persistence: Save/load with compression and integrity validation
- Cross-Platform Builds: Windows (Visual Studio) and Linux (Docker) support
- Production Ready: Stable builds with comprehensive testing
Original implementation with critical issues - DO NOT USE:
- Status: ABANDONED due to build failures and fragmentation problems
- Build Issues: Compilation errors in test files, unstable build system
- Performance Issues: Mammoth dataset splits into scattered pieces, poor manifold preservation
- Missing Features: Lacks AutoHNSW, dual architecture, safety analysis
- Cause: Outdated architecture with random initialization
- Recommendation: Use uwot_umappp_wrapper for all development - legacy version is non-functional
Enhanced production-ready C# wrapper providing .NET integration:
- Enhanced Type-Safe API: Clean C# interface with progress reporting and safety features
- Multi-Dimensional Support: Full API for 1D-50D embeddings
- Distance Metric Selection: Complete enum and validation for all metrics
- Progress Callbacks: .NET delegate integration for real-time feedback
- Safety Features: TransformResult class with outlier detection and confidence scoring
- Cross-Platform: Automatic Windows/Linux runtime detection
- NuGet Ready: Complete package with embedded enhanced native libraries
- Memory Management: Proper IDisposable implementation
- Error Handling: Comprehensive exception mapping from native errors
- Model Information: Rich metadata about fitted models with optimization status
- 1K samples, 50D → 10D: ~200ms
- 10K samples, 100D → 27D: ~2-3 seconds
- 50K samples, 200D → 50D: ~15-20 seconds
- Memory usage: 80-85% reduction vs traditional implementations
- Standard transform: 1-3ms per sample
- Enhanced transform (with safety): 3-5ms per sample
- Batch processing: Near-linear scaling
- Memory: Minimal allocation, production-safe
- Transform Speed: 50-2000x faster than brute force methods
- Memory Usage: 80-85% less than non-optimized implementations
- Accuracy: Identical to reference uwot implementation
- Features: Only implementation with comprehensive safety analysis
- Fixed temp file security vulnerability: Now uses cryptographically secure random generation with proper permissions
- Enhanced LZ4 decompression validation: Added comprehensive bounds checking to prevent buffer overrun attacks
- Complete endian handling: Full cross-platform binary compatibility between Windows/Linux/Mac
- Integer overflow protection: Added safety checks for large dataset allocations
- OpenMP parallelization: Added parallel processing for HNSW point addition (>5000 points)
- Improved K-means convergence: Enhanced convergence detection and empty cluster handling
- L2 normalization fix: Corrected cosine metric normalization for HNSW consistency
- Metric validation warnings: Smart detection of inappropriate data for Hamming (non-binary) and Correlation (constant) metrics
- HNSW reconstruction warnings: Users now get proper notifications when models are loaded from lossy quantized data
- Zero compiler warnings: All unused variables, type conversions, and format issues fixed
- Dead code removal: Eliminated unused functions and cleaned up codebase
- Enhanced error messages: More descriptive error reporting throughout
- Binary version checking: Automatic validation prevents DLL/library version mismatches
- Robust memory management: Improved bounds checking and safe copying operations
- Enhanced test coverage: Comprehensive validation of all error fixes
The fastest way to get started with all enhanced features:
- 🔧 Critical Euclidean Distance Fix: L2Space squared distance now properly converted with sqrt() for exact match detection
- ✅ Perfect Pipeline Consistency: Training embeddings match transform results exactly (MSE = 0)
- 🧪 All Tests Passing: 15/15 C# tests passing (fixed previously failing pipeline tests)
- 🎯 Production Reliability: Proper exact coordinate preservation for identical training points
- 📐 High-Precision Applications: Corrected distance comparisons for validation workflows
Previous v3.15.0 Features:
- 🌊 Stream-based HNSW serialization: Zero temporary files with direct memory-to-file operations
- 🔒 CRC32 data integrity: Automatic corruption detection for both original and embedding space HNSW indices
- ⚡ Deployment-grade reliability: Production-ready model persistence with automatic validation
# Install via NuGet
dotnet add package UMAPuwotSharp --version 3.16.0
# Or clone and build the enhanced C# wrapper
git clone https://github.com/78Spinoza/UMAP.git
cd UMAP/UMAPuwotSharp
dotnet build
dotnet run --project UMAPuwotSharp.Exampleusing UMAPuwotSharp;
Console.WriteLine("=== Enhanced UMAP Demo ===");
// Generate sample data
var data = GenerateTestData(1000, 100);
using var model = new UMapModel();
// Train with progress reporting and custom settings
Console.WriteLine("Training 27D embedding with Cosine metric...");
var embedding = model.FitWithProgress(
data: data,
progressCallback: (epoch, totalEpochs, percent) =>
{
if (epoch % 25 == 0)
Console.WriteLine($" Progress: {percent:F0}% (Epoch {epoch}/{totalEpochs})");
},
embeddingDimension: 27, // High-dimensional embedding
nNeighbors: 20,
minDist: 0.05f,
nEpochs: 300,
metric: DistanceMetric.Cosine // Optimal for high-dim sparse data
);
// Display comprehensive model information
var info = model.ModelInfo;
Console.WriteLine($"\nModel Info: {info}");
Console.WriteLine($" Training samples: {info.TrainingSamples}");
Console.WriteLine($" Input → Output: {info.InputDimension}D → {info.OutputDimension}D");
Console.WriteLine($" Distance metric: {info.MetricName}");
Console.WriteLine($" Neighbors: {info.Neighbors}, Min distance: {info.MinimumDistance}");
// Save enhanced model with HNSW optimization
model.Save("enhanced_model.umap");
Console.WriteLine("Model saved with all enhanced features!");
// Load and transform new data with safety analysis
using var loadedModel = UMapModel.Load("enhanced_model.umap");
var newData = GenerateTestData(100, 100);
// Standard fast transform
var transformedData = loadedModel.Transform(newData);
Console.WriteLine($"Transformed {newData.GetLength(0)} new samples to {transformedData.GetLength(1)}D");
// Enhanced transform with safety analysis
var safetyResults = loadedModel.TransformWithSafety(newData);
var safeCount = safetyResults.Count(r => r.IsProductionReady);
Console.WriteLine($"Safety analysis: {safeCount}/{safetyResults.Length} samples production-ready");🎯 PRODUCTION BUILD: Use the real umappp implementation (CURRENT):
Real umappp build (all advanced features):
cd uwot_umappp_wrapper
BuildWindowsOnly.bat # Windows build
# OR
BuildDockerLinuxWindows.bat # Cross-platform buildThis builds the production-ready umappp implementation with:
- ✅ Spectral initialization: Manifold-aware starting positions (solves mammoth fragmentation!)
- ✅ AutoHNSW optimization: 50-2000x faster with automatic parameter tuning
- ✅ Dual HNSW architecture: Original + embedding space indices for AI inference
- ✅ TransformWithSafety: 5-level outlier detection with confidence scoring
- ✅ Stream-based serialization: CRC32 validation, zero temporary files
- ✅ Complete umappp features: All algorithms from libscran/umappp reference
Legacy build (DEPRECATED - DO NOT USE):
cd uwot_pure_cpp
BuildDockerLinuxWindows.bat # Has compilation errors and build failures❌ CRITICAL: The legacy implementation has build failures, fragmentation issues, and lacks modern optimizations. Use the real umappp implementation for all production applications.
- HNSW optimization: 50-2000x faster transforms with 80-85% memory reduction
- Enhanced algorithms: All new features optimized for performance
- Cross-platform: Windows and Linux support with automatic runtime detection
- Memory efficient: Careful resource management even with high-dimensional embeddings
- Production tested: Comprehensive test suite validating all enhanced functionality including safety features
- 64-bit optimized: Native libraries compiled for x64 architecture with enhanced feature support
- Backward compatible: Models saved with basic features can be loaded by enhanced version
This implementation extends the core C++ algorithms from uwot with:
- HNSW integration: hnswlib for fast approximate nearest neighbor search
- Safety analysis engine: Real-time outlier detection and confidence scoring
- Multi-metric distance computation: Optimized implementations for all five distance metrics
- Arbitrary dimension support: Memory-efficient handling of 1D-50D embeddings
- Progress callback infrastructure: Thread-safe progress reporting from C++ to C#
- Enhanced binary model format: Extended serialization supporting HNSW indices and safety features
- Cross-platform enhanced build system: CMake with Docker support ensuring feature parity
Major Performance & Safety Upgrade! This implementation now includes:
- ⚡ 50-2000x faster transforms with HNSW (Hierarchical Navigable Small World) optimization
- 🛡️ Production safety features - Know if new data is similar to your AI training set
- 📊 Real-time outlier detection with 5-level severity classification
- 🎯 AI model validation - Detect if inference data is "No Man's Land"
- 💾 80% memory reduction for large-scale deployments
- 🔍 Distance-based ML - Use nearest neighbors for classification/regression
Traditional Problem: You train your AI model, but you never know if new inference data is similar to what the model was trained on. This leads to unreliable predictions on out-of-distribution data.
Our Solution: Use UMAP with safety features to validate whether new data points are within the training distribution:
// 1. Train UMAP on your AI training data
var trainData = LoadAITrainingData(); // Your original high-dim data
using var umapModel = new UMapModel();
var embeddings = umapModel.Fit(trainData, embeddingDimension: 10);
// 2. Train your AI model using UMAP embeddings (often better performance)
var aiModel = TrainAIModel(embeddings, labels);
// 3. In production: Validate new inference data
var results = umapModel.TransformWithSafety(newInferenceData);
foreach (var result in results) {
if (result.Severity == OutlierLevel.NoMansLand) {
Console.WriteLine("⚠️ This sample is completely outside training distribution!");
Console.WriteLine(" AI predictions may be unreliable.");
} else if (result.ConfidenceScore > 0.8) {
Console.WriteLine("✅ High confidence - similar to training data");
}
}Use Cases:
- Medical AI: Detect if a new patient's data differs significantly from training cohort
- Financial Models: Identify when market conditions are unlike historical training data
- Computer Vision: Validate if new images are similar to training dataset
- NLP: Detect out-of-domain text that may produce unreliable predictions
- Quality Control: Monitor production data drift over time
Get comprehensive quality analysis for every data point:
var results = model.TransformWithSafety(newData);
foreach (var result in results) {
Console.WriteLine($"Confidence: {result.ConfidenceScore:F3}"); // 0.0-1.0
Console.WriteLine($"Severity: {result.Severity}"); // 5-level classification
Console.WriteLine($"Quality: {result.QualityAssessment}"); // Human-readable
Console.WriteLine($"Production Ready: {result.IsProductionReady}"); // Boolean safety flag
}Safety Levels:
- Normal: Similar to training data (≤95th percentile)
- Unusual: Noteworthy but acceptable (95-99th percentile)
- Mild Outlier: Moderate deviation (99th percentile to 2.5σ)
- Extreme Outlier: Significant deviation (2.5σ to 4σ)
- No Man's Land: Completely outside training distribution (>4σ)
Use nearest neighbor information for additional ML tasks:
var detailedResults = umapModel.TransformDetailed(newData);
foreach (var result in detailedResults) {
// Get indices of k-nearest training samples
var nearestIndices = result.NearestNeighborIndices;
// Use separately saved labels for classification
var nearestLabels = GetLabelsForIndices(nearestIndices);
var predictedClass = nearestLabels.GroupBy(x => x).OrderByDescending(g => g.Count()).First().Key;
// Or weighted regression based on distances
var nearestValues = GetValuesForIndices(nearestIndices);
var weights = result.NearestNeighborDistances.Select(d => 1.0f / (d + 1e-8f));
var predictedValue = WeightedAverage(nearestValues, weights);
Console.WriteLine($"Prediction: {predictedClass} (confidence: {result.ConfidenceScore:F3})");
}Transform Performance (HNSW Optimized):
- Standard transform: 1-3ms per sample
- Enhanced transform (with safety): 3-5ms per sample
- Batch processing: Near-linear scaling
- Memory: 80-85% reduction vs traditional implementations
Comparison vs Other Libraries:
- Training Speed: 50-2000x faster than brute force methods
- Transform Speed: <3ms per sample vs 50-200ms without HNSW
- Memory Usage: 80-85% reduction (15-45MB vs 240MB for large datasets)
- Accuracy: Identical to reference uwot implementation (MSE < 0.01)
- Features: Only C# implementation with HNSW optimization and comprehensive safety analysis
Real-world benchmarks on structured datasets with 3-5 clusters:
| Samples × Features | Exact k-NN | HNSW k-NN | Speedup | Memory Reduction |
|---|---|---|---|---|
| 500 × 25 | 1.2s | 0.6s | 2.0x | 65% |
| 1,000 × 50 | 4.8s | 0.9s | 5.3x | 72% |
| 5,000 × 100 | 2.1 min | 3.2s | 39x | 78% |
| 10,000 × 200 | 12 min | 8.1s | 89x | 82% |
| 20,000 × 300 | 58 min | 18s | 193x | 85% |
| 50,000 × 500 | 6+ hours | 95s | 230x | 87% |
Single sample transform times (after training):
| Dataset Size | Without HNSW | With HNSW | Improvement |
|---|---|---|---|
| 1,000 | 15ms | 2.1ms | 7.1x |
| 5,000 | 89ms | 2.3ms | 38x |
| 20,000 | 178ms | 2.8ms | 64x |
| 100,000 | 890ms | 3.1ms | 287x |
HNSW acceleration works with multiple distance metrics:
| Metric | HNSW Support | Typical Speedup | Best Use Case |
|---|---|---|---|
| Euclidean | ✅ Full | 50-200x | General-purpose data |
| Cosine | ✅ Full | 30-150x | High-dimensional sparse data |
| Manhattan | ✅ Full | 40-180x | Outlier-robust applications |
| Correlation | ⚡ Fallback | 1x (exact) | Time series, correlated features |
| Hamming | ⚡ Fallback | 1x (exact) | Binary, categorical data |
- Minimum: 4GB RAM, dual-core CPU
- Recommended: 8GB+ RAM, quad-core+ CPU with OpenMP
- Optimal: 16GB+ RAM, multi-core CPU with AVX support
Benchmarks performed on Intel i7-10700K (8 cores) with 32GB RAM, Windows 11
- Enhanced Native Libraries: Based on uwot algorithms with revolutionary HNSW optimization
- C# Wrapper: Version 3.42.2+ (UMAPuwotSharp with HNSW optimization)
- Target Framework: .NET 8.0
- Supported Platforms: Windows x64, Linux x64 (both with HNSW optimization)
- Key Features: HNSW k-NN optimization, Production safety, Multi-dimensional (1D-50D), Multi-metric, Enhanced progress reporting, OpenMP parallelization
| Version | Release Date | Key Features | Performance |
|---|---|---|---|
| 3.42.2 | 2024-12-25 | Production-ready code cleanup, Removed debug output, Zero compilation warnings, 17/17 tests passing, Clean deployment | Production-ready deployment |
| 3.42.1 | 2024-12-24 | Fixed single-sample TransformWithSafety, Enhanced fast path with safety metrics (12-15x speedup), Automatic embedding_space_index rebuild for old models, Backward compatibility maintained | Single-sample fix + old model support |
| 3.42.0 | 2024-12-24 | Embedding statistics now calculated (CRITICAL fix), HNSW ordering fixed (nearest-first), AI safety metrics working correctly, Outlier detection operational | Statistics calculation + ordering fix |
| 3.41.0 | 2025-11-07 | Fixed access violation on load, 20x larger file support (100MB→2GB), LZ4 overflow protection, LoadWithCallbacks() API, Fail-fast HNSW validation, Save/load/TransformWithSafety test, Production persistence reliability | Critical load fix + large model support |
| 3.40.0 | 2025-10-31 | InitializationMethod enum (clear API), Spectral default (best quality out-of-the-box), API clarity (replaced boolean with enum), Backward compatibility (obsolete property), Dynamic metadata (model-extracted params), Clean compilation (zero warnings) | Initialization API enhancement |
| 3.39.0 | 2025-10-27 | AlwaysUseSpectral property (force spectral init), Eigen compilation fix (MSVC + performance), LocalConnectivity & Bandwidth integration, Bandwidth sweep testing, Enhanced visualization titles, Spectral quality boost | Advanced algorithms revolution |
| 3.37.0 | 2025-10-26 | OpenMP parallelization (4-5x transform speedup), Single-point optimization (12-15x speedup), Stringstream persistence (no temp files), Windows DLL stability, Thread-safe operations | Major performance revolution |
| 3.33.1 | 2025-10-25 | Dual-mode exact k-NN integration, CPU core reporting, Complete parameter propagation, umappp with knncolle, Enhanced progress monitoring | Production-grade dual-mode accuracy |
| 3.16.0 | 2025-10-02 | Critical Euclidean distance fix, Perfect pipeline consistency (MSE=0), All 15/15 tests passing, Exact coordinate preservation | Production reliability fix |
| 3.15.0 | 2025-02-02 | Stream-based HNSW serialization, CRC32 data integrity validation, Zero temporary files, Enhanced test thresholds | Deployment-grade reliability |
| 3.14.0 | 2025-02-01 | Dual HNSW architecture, AI pattern similarity search, Embedding space inference, 5-level outlier detection | Revolutionary AI capabilities |
| 3.3.0 | 2025-01-22 | Enhanced HNSW optimization, Improved memory efficiency, Better progress reporting, Cross-platform stability | Refined HNSW performance |
| 3.1.2 | 2025-01-15 | Smart spread parameter implementation, Dimension-aware defaults, Enhanced progress reporting | Optimal embedding quality across dimensions |
| 3.1.0 | 2025-01-15 | Revolutionary HNSW optimization, Enhanced API with forceExactKnn parameter, Multi-core OpenMP acceleration | 50-2000x speedup, 80-85% memory reduction |
| 3.0.1 | 2025-01-10 | Critical cross-platform fix, Linux HNSW library (174KB), Enhanced build system | Full cross-platform HNSW parity |
| 3.0.0 | 2025-01-08 | First HNSW implementation, Production safety features, 5-level outlier detection | 50-200x speedup (Windows only) |
| 2.x | 2024-12-XX | Standard UMAP implementation, Multi-dimensional support (1D-50D), Multi-metric, Progress reporting | Traditional O(n²) performance |
// v2.x code (still supported)
var embedding = model.Fit(data, embeddingDimension: 2);
// v3.33.1 optimized code - Dual-mode exact k-NN + CPU core reporting
var fastEmbedding = model.Fit(data,
embeddingDimension: 2,
forceExactKnn: false, // HNSW mode (default) - 50-2000x faster
useQuantization: true, // Enable 85-95% compression
progressCallback: progress => Console.WriteLine($"Progress: {progress.PercentComplete:F1}% - {progress.Message}")
// Shows: "Using 16 CPU cores for parallel processing"
);
// Exact k-NN mode for research/validation
var exactEmbedding = model.Fit(data,
embeddingDimension: 2,
forceExactKnn: true, // Exact k-NN mode - perfect accuracy
progressCallback: progress => Console.WriteLine($"Progress: {progress.PercentComplete:F1}% - {progress.Message}")
// Shows: "force_exact_knn = true - using exact k-NN computation"
);
// Model persistence includes automatic CRC32 validation
model.SaveModel("model.umap"); // Stream-based serialization with integrity checks
var loadedModel = UMapModel.LoadModel("model.umap"); // Automatic corruption detectionRecommendation: Upgrade to v3.42.2 for production-ready code with fully working single-sample TransformWithSafety, automatic old model support, and clean deployment with zero debug output.
- McInnes, L., Healy, J., & Melville, J. (2018). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426.
- Malkov, Yu A., and D. A. Yashunin. "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs." arXiv:1603.09320 (2018).
- Interactive UMAP Guide: https://pair-code.github.io/understanding-umap/
- uwot R package: https://github.com/jlmelville/uwot
- hnswlib library: https://github.com/nmslib/hnswlib
- Original Python UMAP: https://github.com/lmcinnes/umap
Maintains compatibility with the GPL-3 license of the original uwot package and Apache 2.0 license of hnswlib.
This enhanced implementation represents the most complete and feature-rich UMAP library available for C#/.NET, providing capabilities that surpass even many Python implementations. The combination of HNSW optimization, production safety features, arbitrary embedding dimensions, multiple distance metrics, progress reporting, and complete model persistence makes it ideal for both research and production machine learning applications.