From c26dd15d8ff375f283bc323ada0cbb1447c220b4 Mon Sep 17 00:00:00 2001 From: qrcodebusiness07 Date: Sun, 28 Dec 2025 12:52:22 +0500 Subject: [PATCH 1/8] docs: add d2d options and capabilities reference page Signed-off-by: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> --- docs/d2d-options.rst | 13 +++++++++++++ docs/index.rst | 2 ++ 2 files changed, 15 insertions(+) create mode 100644 docs/d2d-options.rst diff --git a/docs/d2d-options.rst b/docs/d2d-options.rst new file mode 100644 index 0000000000..e23b12c93b --- /dev/null +++ b/docs/d2d-options.rst @@ -0,0 +1,13 @@ +Reference Documentation Page: d2d options and capabilities +========================================================= + +This page lists the available d2d options and highlights the capabilities present for each optional ecosystem option. + +Capabilities matrix +------------------- + ++------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ +| d2d option | compiled file -> source file | archive -> source directory | binary/source symbols | archives -> purlDB | source files -> purlDB | ++==================+==============================+==============================+===========================+=======================+==========================+ +| TODO | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | ++------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ diff --git a/docs/index.rst b/docs/index.rst index d1ce2299c6..720ba64b64 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -57,6 +57,8 @@ In this documentation, you’ll find: webhooks application-settings distros-os-images + d2d-options + Indices and tables ================== From c23ce4357baf52faf1af779491e1eafe3ab638b1 Mon Sep 17 00:00:00 2001 From: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> Date: Sun, 28 Dec 2025 13:11:23 +0500 Subject: [PATCH 2/8] docs: complete d2d options reference page Signed-off-by: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> --- docs/d2d-options.rst | 90 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 85 insertions(+), 5 deletions(-) diff --git a/docs/d2d-options.rst b/docs/d2d-options.rst index e23b12c93b..bb3dbcdcae 100644 --- a/docs/d2d-options.rst +++ b/docs/d2d-options.rst @@ -3,11 +3,91 @@ Reference Documentation Page: d2d options and capabilities This page lists the available d2d options and highlights the capabilities present for each optional ecosystem option. +Overview +-------- + +d2d (deploy-to-devel) workflows are used to map deployed artifacts (such as compiled binaries and archives) +back to their most likely source code. + +The current d2d helper script runs the ``map_deploy_to_develop`` mapping workflow in ScanCode.io inside a +Docker container. It can optionally spin up a temporary PostgreSQL database when needed. + +Usage +----- + +.. code-block:: bash + + ./map-deploy-to-develop.sh [options] [db-port] + +Arguments +--------- + ++-----------------+-------------------------------------------------------------+ +| Argument | Description | ++=================+=============================================================+ +| ``from-path`` | Path to the base deployment/scan file | ++-----------------+-------------------------------------------------------------+ +| ``to-path`` | Path to the target deployment/scan file | ++-----------------+-------------------------------------------------------------+ +| ``options`` | D2D pipeline parameters (can be empty ``""``) | ++-----------------+-------------------------------------------------------------+ +| ``output-file`` | File where ScanCode.io output will be written | ++-----------------+-------------------------------------------------------------+ +| ``spin-db`` | ``true`` = spin temp DB container, ``false`` = skip | ++-----------------+-------------------------------------------------------------+ +| ``db-port`` | Port to bind Postgres (default: ``5432``) | ++-----------------+-------------------------------------------------------------+ + Capabilities matrix ------------------- -+------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ -| d2d option | compiled file -> source file | archive -> source directory | binary/source symbols | archives -> purlDB | source files -> purlDB | -+==================+==============================+==============================+===========================+=======================+==========================+ -| TODO | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | -+------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ +The helper script executes the ``map_deploy_to_develop`` mapping workflow. The exact capabilities depend on the +D2D pipeline parameters passed through the ``options`` argument. + ++------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ +| d2d option / parameter | compiled file -> source file | archive -> source directory | binary/source symbols | archives -> purlDB | source files -> purlDB | ++==========================================+==============================+==============================+===========================+=======================+==========================+ +| Base mapping workflow (default) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | ++------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ +| ``options`` (pipeline parameters) | ✅ | ✅ | ✅/❌ | ✅/❌ | ✅/❌ | ++------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ +| ``spin-db=true`` (temporary DB enabled) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | ++------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ +| ``spin-db=false`` (no DB container) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | ++------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ +| ``db-port=`` | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | ++------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ + +Examples +-------- + +Run mapping without database: + +.. code-block:: bash + + ./map-deploy-to-develop.sh ./from.tar.gz ./to.whl results.json "" false + +Run mapping with database on a custom port: + +.. code-block:: bash + + ./map-deploy-to-develop.sh ./from.tar.gz ./to.whl output.json "Python,Java" true 5433 + +Script actions (high-level) +--------------------------- + +1. Validates required arguments +2. Starts PostgreSQL in Docker (if ``spin-db=true``) +3. Creates a temporary working directory: ``./d2d`` +4. Copies input files into working directory +5. Runs ScanCode.io mapping step +6. Writes mapping output into ``output-file`` +7. Cleans up temp directory +8. Stops DB container if it was started + +Related files +------------- + +- Script: ``etc/scripts/d2d/map-deploy-to-develop.sh`` +- README: ``etc/scripts/d2d/README.rst`` +- Reference docs index: :doc:`index` From ce1924b300c5c1ba7ae947c4f8d540bcec0b7ffa Mon Sep 17 00:00:00 2001 From: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> Date: Mon, 29 Dec 2025 18:32:09 +0500 Subject: [PATCH 3/8] Update docs/d2d-options.rst Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> --- docs/d2d-options.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/d2d-options.rst b/docs/d2d-options.rst index bb3dbcdcae..bc636722af 100644 --- a/docs/d2d-options.rst +++ b/docs/d2d-options.rst @@ -1,4 +1,4 @@ -Reference Documentation Page: d2d options and capabilities +Reference Documentation Page: D2D Options and Capabilities ========================================================= This page lists the available d2d options and highlights the capabilities present for each optional ecosystem option. From 1104ac9ae1f6b9377444667616ca8a185a7ef63e Mon Sep 17 00:00:00 2001 From: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> Date: Mon, 29 Dec 2025 18:32:26 +0500 Subject: [PATCH 4/8] Update docs/d2d-options.rst Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> --- docs/d2d-options.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/d2d-options.rst b/docs/d2d-options.rst index bc636722af..0aadba4503 100644 --- a/docs/d2d-options.rst +++ b/docs/d2d-options.rst @@ -6,7 +6,7 @@ This page lists the available d2d options and highlights the capabilities presen Overview -------- -d2d (deploy-to-devel) workflows are used to map deployed artifacts (such as compiled binaries and archives) +d2d (deploy-to-develop) workflows are used to map deployed artifacts (such as compiled binaries and archives) back to their most likely source code. The current d2d helper script runs the ``map_deploy_to_develop`` mapping workflow in ScanCode.io inside a From 62f948b6d9a4a0f99d3680015ec6481cd9b99d85 Mon Sep 17 00:00:00 2001 From: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> Date: Mon, 29 Dec 2025 18:33:37 +0500 Subject: [PATCH 5/8] Update docs/d2d-options.rst Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> --- docs/d2d-options.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/d2d-options.rst b/docs/d2d-options.rst index 0aadba4503..c7c367f225 100644 --- a/docs/d2d-options.rst +++ b/docs/d2d-options.rst @@ -35,7 +35,7 @@ Arguments +-----------------+-------------------------------------------------------------+ | ``spin-db`` | ``true`` = spin temp DB container, ``false`` = skip | +-----------------+-------------------------------------------------------------+ -| ``db-port`` | Port to bind Postgres (default: ``5432``) | +| ``db-port`` | ``--port`` flag to specify port to bind Postgres (default: ``5432``) | +-----------------+-------------------------------------------------------------+ Capabilities matrix From 37feee649838b464f9270fcd92a13c13f08a9cea Mon Sep 17 00:00:00 2001 From: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> Date: Mon, 29 Dec 2025 18:33:51 +0500 Subject: [PATCH 6/8] Update docs/d2d-options.rst Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> --- docs/d2d-options.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/d2d-options.rst b/docs/d2d-options.rst index c7c367f225..25ee3dee88 100644 --- a/docs/d2d-options.rst +++ b/docs/d2d-options.rst @@ -65,7 +65,7 @@ Run mapping without database: .. code-block:: bash - ./map-deploy-to-develop.sh ./from.tar.gz ./to.whl results.json "" false + ./map-deploy-to-develop.sh ./from.tar.gz ./to.whl results.json Run mapping with database on a custom port: From e56bb37180cb94d7ee9c6230dfe6f1f9fbcf4d38 Mon Sep 17 00:00:00 2001 From: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> Date: Mon, 29 Dec 2025 21:25:51 +0500 Subject: [PATCH 7/8] Fix d2d docs: flags/usage/examples Signed-off-by: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> --- docs/d2d-options.rst | 74 ++++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 33 deletions(-) diff --git a/docs/d2d-options.rst b/docs/d2d-options.rst index 25ee3dee88..a9b589523b 100644 --- a/docs/d2d-options.rst +++ b/docs/d2d-options.rst @@ -17,46 +17,52 @@ Usage .. code-block:: bash - ./map-deploy-to-develop.sh [options] [db-port] + ./map-deploy-to-develop.sh [--options ] [--spin-db] [--port ] + Arguments --------- -+-----------------+-------------------------------------------------------------+ -| Argument | Description | -+=================+=============================================================+ -| ``from-path`` | Path to the base deployment/scan file | -+-----------------+-------------------------------------------------------------+ -| ``to-path`` | Path to the target deployment/scan file | -+-----------------+-------------------------------------------------------------+ -| ``options`` | D2D pipeline parameters (can be empty ``""``) | -+-----------------+-------------------------------------------------------------+ -| ``output-file`` | File where ScanCode.io output will be written | -+-----------------+-------------------------------------------------------------+ -| ``spin-db`` | ``true`` = spin temp DB container, ``false`` = skip | -+-----------------+-------------------------------------------------------------+ -| ``db-port`` | ``--port`` flag to specify port to bind Postgres (default: ``5432``) | -+-----------------+-------------------------------------------------------------+ ++-----------------+---------------------------------------------------------------------+ +| Argument | Description | ++=================+=====================================================================+ +| ``from-path`` | Path to the base deployment/scan file | ++-----------------+---------------------------------------------------------------------+ +| ``to-path`` | Path to the target deployment/scan file | ++-----------------+---------------------------------------------------------------------+ +| ``output-file`` | File where ScanCode.io output will be written | ++-----------------+---------------------------------------------------------------------+ +| ``--options`` | Optional flag to specify D2D pipeline parameters | +| | (for example: ``"Python,Java"``). | ++-----------------+---------------------------------------------------------------------+ +| ``--spin-db`` | Optional flag to start a temporary PostgreSQL DB container | +| | (omit the flag to skip starting the DB). | ++-----------------+---------------------------------------------------------------------+ +| ``--port`` | Optional flag to specify the Postgres bind port | +| | (default: ``5432``). | ++-----------------+---------------------------------------------------------------------+ + Capabilities matrix ------------------- The helper script executes the ``map_deploy_to_develop`` mapping workflow. The exact capabilities depend on the -D2D pipeline parameters passed through the ``options`` argument. - -+------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ -| d2d option / parameter | compiled file -> source file | archive -> source directory | binary/source symbols | archives -> purlDB | source files -> purlDB | -+==========================================+==============================+==============================+===========================+=======================+==========================+ -| Base mapping workflow (default) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | -+------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ -| ``options`` (pipeline parameters) | ✅ | ✅ | ✅/❌ | ✅/❌ | ✅/❌ | -+------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ -| ``spin-db=true`` (temporary DB enabled) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | -+------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ -| ``spin-db=false`` (no DB container) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | -+------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ -| ``db-port=`` | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | -+------------------------------------------+------------------------------+------------------------------+---------------------------+-----------------------+--------------------------+ +D2D pipeline parameters passed through the ``--options`` flag. + ++--------------------------------------------+------------------------------+------------------------------+---------------------------+------------------------+--------------------------+ +| d2d option / parameter | compiled file -> source file | archive -> source directory | binary/source symbols | archives -> purlDB | source files -> purlDB | ++============================================+==============================+==============================+===========================+========================+==========================+ +| Base mapping workflow (default) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | ++--------------------------------------------+------------------------------+------------------------------+---------------------------+------------------------+--------------------------+ +| ``--options `` (pipeline parameters)| ✅ | ✅ | ✅/❌ | ✅/❌ | ✅/❌ | ++--------------------------------------------+------------------------------+------------------------------+---------------------------+------------------------+--------------------------+ +| ``--spin-db`` (temporary DB enabled) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | ++--------------------------------------------+------------------------------+------------------------------+---------------------------+------------------------+--------------------------+ +| without ``--spin-db`` (no DB container) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | ++--------------------------------------------+------------------------------+------------------------------+---------------------------+------------------------+--------------------------+ +| ``--port `` | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | ++--------------------------------------------+------------------------------+------------------------------+---------------------------+------------------------+--------------------------+ + Examples -------- @@ -71,13 +77,14 @@ Run mapping with database on a custom port: .. code-block:: bash - ./map-deploy-to-develop.sh ./from.tar.gz ./to.whl output.json "Python,Java" true 5433 + ./map-deploy-to-develop.sh ./from.tar.gz ./to.whl output.json --options "Python,Java" --spin-db --port 5433 + Script actions (high-level) --------------------------- 1. Validates required arguments -2. Starts PostgreSQL in Docker (if ``spin-db=true``) +2. Starts PostgreSQL in Docker (if ``--spin-db`` is provided) 3. Creates a temporary working directory: ``./d2d`` 4. Copies input files into working directory 5. Runs ScanCode.io mapping step @@ -85,6 +92,7 @@ Script actions (high-level) 7. Cleans up temp directory 8. Stops DB container if it was started + Related files ------------- From b1c2e331921f2c1027873a674a8a84d3a689a6fe Mon Sep 17 00:00:00 2001 From: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> Date: Mon, 29 Dec 2025 21:35:35 +0500 Subject: [PATCH 8/8] Fix d2d docs: correct flags, args order, examples Signed-off-by: Muhammad Abdullah Shahid <108458621+abdollahShahid@users.noreply.github.com> --- docs/d2d-options.rst | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/d2d-options.rst b/docs/d2d-options.rst index a9b589523b..984d60925e 100644 --- a/docs/d2d-options.rst +++ b/docs/d2d-options.rst @@ -17,8 +17,8 @@ Usage .. code-block:: bash - ./map-deploy-to-develop.sh [--options ] [--spin-db] [--port ] - + ./map-deploy-to-develop.sh \ + [--options ] [--spin-db] [--db-port ] Arguments --------- @@ -38,11 +38,10 @@ Arguments | ``--spin-db`` | Optional flag to start a temporary PostgreSQL DB container | | | (omit the flag to skip starting the DB). | +-----------------+---------------------------------------------------------------------+ -| ``--port`` | Optional flag to specify the Postgres bind port | +| ``--db-port`` | Optional flag to specify the Postgres bind port | | | (default: ``5432``). | +-----------------+---------------------------------------------------------------------+ - Capabilities matrix ------------------- @@ -60,10 +59,9 @@ D2D pipeline parameters passed through the ``--options`` flag. +--------------------------------------------+------------------------------+------------------------------+---------------------------+------------------------+--------------------------+ | without ``--spin-db`` (no DB container) | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | +--------------------------------------------+------------------------------+------------------------------+---------------------------+------------------------+--------------------------+ -| ``--port `` | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | +| ``--db-port `` | ✅ | ✅ | depends on parameters | depends on parameters | depends on parameters | +--------------------------------------------+------------------------------+------------------------------+---------------------------+------------------------+--------------------------+ - Examples -------- @@ -77,8 +75,10 @@ Run mapping with database on a custom port: .. code-block:: bash - ./map-deploy-to-develop.sh ./from.tar.gz ./to.whl output.json --options "Python,Java" --spin-db --port 5433 - + ./map-deploy-to-develop.sh ./from.tar.gz ./to.whl output.json \ + --options "Python,Java" \ + --spin-db \ + --db-port 5433 Script actions (high-level) --------------------------- @@ -92,7 +92,6 @@ Script actions (high-level) 7. Cleans up temp directory 8. Stops DB container if it was started - Related files -------------