Merging SBOMs doesn’t get rid of duplicates

[yaourabi]

I have a Gradle multi-project build, in which i generate an SBOM for java dependencies for each project using the CycloneDX Gradle plugin. I want to merge all the existing SBOMs, however the resulting SBOM doesn’t get rid of duplicated dependencies.

Steps to reproduce

• Create a build with two sub-projects.

dependencies for the first project

dependencies {
    implementation("org.apache.commons:commons-lang3:3.12.0")
}

dependencies for the second project

dependencies {
    implementation("org.apache.commons:commons-lang3:3.12.0")
    implementation("com.fasterxml.jackson.core:jackson-databind:2.15.3")
}

• Apply the CycloneDX Gradle plugin for each project

tasks.cyclonedxBom {
    setProjectType("application")
    setSchemaVersion("1.5")
    setDestination(project.file("build/reports"))
    setOutputName("bom2")
    setOutputFormat("json")
    setIncludeBomSerialNumber(false)
    setIncludeLicenseText(true)
    setComponentVersion("2.0.0")
}

• Generate an SBOM for each project
• Use the cyclonedx-cli merge command to merge the two SBOMs

CycloneDX cli version: 0.25.0 0.25.0

CycloneDX Gradle plugin version: 1.7.4

Expected behavior

although the commons-lang3 dependency is defined in both projects, I only want it to be declared in the final SBOM once.

Current behavior

the current SBOM declares the commons-lang3 dependency twice.

"dependencies": [
    {
      "ref": "pkg:maven/org.apache.commons/commons-lang3@3.12.0?type=jar",
      "dependsOn": []
    },
    {
      "ref": "pkg:maven/org.example/app1@1.0-SNAPSHOT?type=jar",
      "dependsOn": [
        "pkg:maven/org.apache.commons/commons-lang3@3.12.0?type=jar"
      ]
    },
    {
      "ref": "pkg:maven/com.fasterxml.jackson.core/jackson-annotations@2.15.3?type=jar",
      "dependsOn": []
    },
    {
      "ref": "pkg:maven/org.apache.commons/commons-lang3@3.12.0?type=jar",
      "dependsOn": []
    },
    {
      "ref": "pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.15.3?type=jar",
      "dependsOn": [
        "pkg:maven/com.fasterxml.jackson.core/jackson-annotations@2.15.3?type=jar",
        "pkg:maven/com.fasterxml.jackson.core/jackson-core@2.15.3?type=jar"
      ]
    },
    {
      "ref": "pkg:maven/org.example/app2@1.0-SNAPSHOT?type=jar",
      "dependsOn": [
        "pkg:maven/org.apache.commons/commons-lang3@3.12.0?type=jar",
        "pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.15.3?type=jar"
      ]
    },
    {
      "ref": "pkg:maven/com.fasterxml.jackson.core/jackson-core@2.15.3?type=jar",
      "dependsOn": []
    }
  ]

[ertl]

+1

[zabulus]

+1. This blows up count of both components and vulnerabilities in dependency-track.
Use-case is following:

1. We create sbom file for each .csproj separately using the cyclonedx dotnet tool.
2. Then we merge everything in single file using the cyclonedx cli tool.
3. Import resulting file to dependency-track, see over 10k components with lots of duplicates in project

[ferben]

+1
Some --deduplicate option would be very useful 👍

[msherms2]

+1 Similar flow, we have individual components but also a global "the world" release level component that dependency resolution may be helpful at. Foreach SBOM merged in I see a separate Component.

[dpkano]

+1. We're a year in this now. Is there any update on this track?

[mtsfoni]

+1 I'd also like to see this fixed.

[aniziki]

+1 Any update on this?

[AngeloFilaseta]

+1

[karimaazergui]

+1

[andreas-hilti]

@yaourabi Can you please attach minimal BOMs as well as the resulting BOM?

When the components are exactly identical, then they are filtered out in a flat merge, both in the components list, as well as in the dependencies list; see this corresponding test:
CycloneDX/cyclonedx-dotnet-library#416

I'd like to understand what is missed in this test; maybe they differ slightly, but in which way?