Duplicate package-urls may lead to unmapped components in `bom map`

[tngraf]

Duplicate package-urls - more precise duplicate package-urls in different components - may lead to unmapped components in bom map.

Example of a failed bom map:

...
 jackson-core, 2.17.2
    Releases purls point to different components: {'https://sw360.dummy.com/resource/api/components/df946d849687b79d1348e0b09a0ae101': '2.18.0', 'https://sw360.dummy.com/resource/api/components/95459e7828d44df4b110239b74587aea': '2.5.0'}
...
Mapping result:
  ...
  No match, jackson-core, 2.17.2 
...
Total releases    = 33
  Full matches    = 32
  Name matches    = 0
  Similar matches = 0
  No match        = 1
 Creating result overview overview.json
...
No unique mapping found - manual action needed!

It does not to be exactly the same package-url. In the example above one is pkg:maven/com.fasterxml.jackson.core/jackson-core@2.18.0?type=jar and the other one is pkg:maven/com.fasterxml.jackson.core/jackson-core@2.5.0?type=jar.

[gernot-h]

@tngraf, do you expect me to have a look here? I still have #122 on my TODO list, but after this, I could also have a look into this issue or did you already start digging deeper here?

[Sadaf-A]

@tngraf I would like to work on this could you please assign?

[gernot-h]

@Sadaf-A, do you have an idea for a good approach here? If yes, feel free to outline your idea here! That would probably be better than starting preparing a PR as I think we need to agree on a good approach first.

I didn't really analyze it yet, but I think the problem comes from a combination of failed purl-based component search and failed name based component search. Purl-based search fails due to releases of different components sharing the same purl component name and name-based search fails due to spelling differences. So I've no good idea yet how a canonical solution should look like.

[SUMITKC1]

Hi @tngraf,

I have analyzed the issue and would like to propose an approach to handle duplicate package-URLs effectively in the BOM mapping process.
🔹 Proposed Solution
✔ Canonical Mapping Approach:

Instead of treating each package-URL as unique, maintain a mapping of all purls pointing to multiple possible versions.

If a single package-URL maps to multiple components, identify the best match based on semantic versioning and component hierarchy.

✔ Fallback Matching Strategy:

If purl-based mapping fails, attempt fuzzy matching on component names (e.g., Levenshtein distance or Jaccard similarity).

Prioritize components with similar names and close version numbers.

✔ Enhanced Logging & Debugging:

Log ambiguous mappings and suggest potential manual resolutions instead of failing completely.

Store intermediate matching results in overview.json for easier debugging.

🛠 Implementation Plan
1️⃣ Refactor BOM Mapping Logic:

Introduce a dictionary-based purl tracker to handle duplicates efficiently.

Implement rules for resolving multiple matches based on versioning and component metadata.

2️⃣ Improve Fuzzy Matching for Name-Based Search:

Use string similarity metrics to improve accuracy in case of slight spelling differences.

3️⃣ Modify Logging & Debugging:

Instead of stopping at "No unique mapping found", provide a ranked list of potential matches.

Would you be open to a dictionary-based mapping approach with fuzzy fallback matching?

If this approach seems reasonable, I will start implementing it and submit a PR.

Let me know if you have any feedback or alternative suggestions!

Thanks! 🚀

[gernot-h]

Dear @SUMITKC1, thanks for your suggestions! I have one question:

Instead of treating each package-URL as unique, maintain a mapping of all purls pointing to multiple possible versions.

If a single package-URL maps to multiple components, identify the best match based on semantic versioning and component hierarchy.

Hmm interesting. Can you please elaborate this strategy? What do you mean by "based on semantic versioning and component hierarchy"?

✔ Fallback Matching Strategy:

If purl-based mapping fails, attempt fuzzy matching on component names (e.g., Levenshtein distance or Jaccard similarity).

Prioritize components with similar names and close version numbers.
[...]
Instead of stopping at "No unique mapping found", provide a ranked list of potential matches.

Hmm, that's something where I would be interested in @tngraf's opinion. In general, I'm no friend of more "fuzzy matching" strategies as I see the risk that users would just automate taking the best match without manual review. And I've seen just too many cases where even as an experienced, long-term Linux user it requires more than a short look to identify if a match is correct or not. For me, such an approach seems more feasible for an interactive tool where the user can immediately check multiple candidates and decide between them, but not really for a CLI batch tool as CaPyCli. But this is just my opinion...

[SUMITKC1]

Hi @gernot-h,

Thanks for your feedback! I see your concern about automated fuzzy matching, and I’d like to refine my approach based on that to ensure accuracy and maintainability.

Clarification on Semantic Versioning & Component Hierarchy

What I meant by this is:

• When multiple components share the same package-URL, we can apply semantic versioning rules (e.g., preferring the closest stable version or minor version match).
• Component hierarchy refers to identifying the correct component based on dependencies or usage context—helping avoid mismatches while maintaining accuracy.

Re-evaluating Fuzzy Matching

I completely understand the risks of incorrect automated matches. Instead of full automation, I suggest:

• Logging possible matches instead of auto-selecting – Users can review them manually before proceeding.
• Adding an optional interactive mode – This would allow users to select from ranked matches in cases where interaction is possible.

Would this approach work? If so, I’d love to start implementing it and submit a PR for review. Let me know your thoughts, and I’d be happy to refine things further if needed! 🚀

[gernot-h]

Have looked at this in a bit more detail, also with an additional example @tngraf sent me via chat. The current code basically considers the PURL to allow for full automatic processing of an SBOM, i.e.:

• a release PURL shall only point to a single release
• a component PURL shall only point to a single component. If there's no component PURL found, release purls will be searched and all release purls sharing the same component name shall point to releases within a single component.

Otherwise, automatic mapping or creation of releases won't work as the code wouldn't know which component or release to use.

Therefore, the current PURL cache is designed to only return one release or one component if asked for a PURL. If it finds duplicates, the corresponding entries won't be stored in the cache (and a warning would be printed, but this is usually disabled).

So I think this issue describes several problems:

• If multiple releases have the same PURL pkg:pypi/foo@1.0, nothing will be found by id when searching for pkg:pypi/foo@1.0. I think this is the example @tngraf shared with me via chat.
• If componentA has a release with pkg:pypi/foo@0.1 and componentB has a release with pkg:pypi/foo@0.2, no component will be found by id when searching pkg:pypi/foo@0.3. I think this is what is the description of this issue is showing. Search for existing, unique PURLs like foo@0.1 should however work (untested).
• In all cases, the component and versions shall still be found by string search if the spelling is identical. We have no real fuzzy matching as suggested in the comments above.

Personally, I'm no fan of relaxing the assumptions about PURLs - in my eyes, the goal of using PURLs is to allow for automation. So if purl search won't provide a result due to duplicates, someone should fix the SW360 entries instead of also introducing support for duplicates in the PURL search.

What however might make sense is to store information about duplicates silently, but inform the user that an entry was not found due to inconsistent data.

Whether we want to add fuzzy name search is a different topic, not sure however how much effort we want to invest and how much complexity we want to allow here.

So I guess the general approach needs to be discussed with @tngraf before we can decide how to proceed here. And we should have test cases verifying our assumptions. :-)

[SUMITKC1]

Hi @gernot-h,

Thanks for the detailed breakdown! I now see that PURLs are designed for strict automation, and handling duplicates within the search may not be ideal.

Would it make sense to:

Log duplicate PURLs silently while informing users of inconsistent data?

Validate assumptions with test cases to avoid unintended behavior?

Discuss with @tngraf whether fuzzy name search is worth exploring separately?

I’d love to contribute—would you be open to me working on duplicate handling and test improvements? Let me know how you'd like to proceed! 🚀

[gernot-h]

[...]

Discuss with @tngraf whether fuzzy name search is worth exploring separately?

I’d love to contribute—would you be open to me working on duplicate handling and test improvements? Let me know how you'd like to proceed! 🚀

@SUMITKC1, I think @tngraf and me need to work on this topic first, agree on the approach and adapt the code accordingly before it makes sense for you to contribute with possible improvements. I would also prefer to work on one topic at a time, so let's concentrate on #121 for now.

[tngraf]

Dear all,
I see no advantage in any more elaborate matching strategy.
The issue we are discussing here, is that we have a gap, when there are duplicate purls. This is the only thing that needs to get solved, because we we would cleary miss an exisitng match.

Fuzzy matching or levenshtein distance do not provide a better match. It is then just better top tell the user that there is no (automated) match. Then the user can think about a better match ... or about better SBOM data.

[gernot-h]

Thanks, @tngraf, for the clarification! I agree with you that making name/version-based search more fuzzy isn't the right direction.

But if I'm not mistaken, the question is now: We have no mapping result with name/version search and we have ambiguous mapping results with the package identifier (which has the goal to provide unambiguous results!). So what shall we do...

• if there is no matching release, but we find multiple matching components?
• if there are multiple matching releases?

In my eyes, we shall in neither of these cases create a BOM which has multiple "good matches" with entries with -full-match- as downstream scripts might rely on good matches to be unambiguous. What we could do is add new types of mapping results for this situation, e.g. 7-candidate-match-by-release-id, 8-candidate-match-by-component-id. This however would mean a larger refactoring of the existing code.

And now, the next question is: What will users do with these mapping results? Will they carefully review the output BOM, then check all candidate matches in SW360 to decide which one is the better one and then use it? Or will they just select a random one of these entries and use it?

For me, candidate matches are not really useful in general. If there's no good match for an SBOM item, I need to do manual work anyways, so I'll directly go to SW360 and check for possible candidates applying my common sense. Who says that entries which have ambiguous package URLs (showing their creators used broken workflows) are better than those with slightly different spelling overlooked by the traditional name-based search?

[SUMITKC1]

Hi @gernot-h, @tngraf,

Thank you both for the detailed discussion and for outlining the architectural intent behind the current mapping logic. I now clearly understand why fuzzy matching and duplicate-resolution strategies are being avoided in favor of clarity and strict automation.

I'll hold off on submitting any PR here until you've finalized the design direction for handling these edge cases internally. Once things are clearer or if smaller contributions (like improved logging or test coverage) would be helpful in this context, I'd love to support.

In the meantime, I’ll stay focused on issue #121 as suggested. Thanks again for your feedback and guidance!

[gernot-h]

I started reworking the PURL cache to allow for storing duplicate PURLs, see #143, as this is needed for this issue as well as for #130.