Fix: deterministic multithreaded lidar odometry#377
Closed
bloom256 wants to merge 4 commits intoMapsHD:mainfrom
Closed
Fix: deterministic multithreaded lidar odometry#377bloom256 wants to merge 4 commits intoMapsHD:mainfrom
bloom256 wants to merge 4 commits intoMapsHD:mainfrom
Conversation
…eterministic multithreaded Hessian accumulation
…order sequential summation, making ST and MT paths similar and deterministic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
optimize_lidar_odometrygives different results between multithreaded runs.The multithreaded path used
tbb::combinable+combine_eachto add upHessian matrices from per-thread local copies.
combine_eachgoes throughthread-local storage in an unspecified order, and since floating-point addition
is not associative, different summation orders give different Hessian matrices.
Over many optimiser iterations, these small differences grow and can lead to
different convergence paths.
Fix
Replace
tbb::combinablewith per-pointAtPA/AtPBstorage. Each point's6x6 and 6x1 Hessian contributions are computed in parallel (via
tbb::parallel_forover points), then summed sequentially in fixed point order similar for ST and MT.The ST and MT paths now share the same code -- only the loop type differs
(
tbb::parallel_forvs plainfor).Changes in
lidar_odometry_utils_optimizers.cpp#include <tbb/combinable.h>and#include <tbb/blocked_range.h>add_indoor_hessian_contribution/add_outdoor_hessian_contributionnowwrite to per-point output matrices instead of accumulating into the global
Hessian directly
compute_hessiantakes per-point vectors + point index instead of globalmatrices
sequential), sum sequentially in point order
UTL_PROFILER_END(before_iter)that was insideprocess_worker_step_lidar_odometry_corebut its matchingBEGINwas in thecaller -- moved it back to the caller
UTL_PROFILER_SCOPEtoprocess_worker_step_1,process_worker_step_2,and
process_worker_step_lidar_odometry_coreNote: This was not caught in the previous nondeterminism PR because the
floating-point differences are very small and only showed up on certain datasets
during extended testing.
Testing
Tested on 7 datasets (4 MT + 4 ST runs each), lengths from 500m to 5000m.
All runs showed deterministic behaviour, with identical results between ST and MT.