[Datasets] Task/rdmp 33 dataset integration interface #2182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft

JFriel wants to merge 165 commits into feature/datasets from task/RDMP-33-dataset-integration-interface

.github/workflows/build.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -17,9 +17,9 @@ jobs:
  
        runs-on: windows-latest

        steps:

          - name: Checkout code

            uses: actions/checkout@v4

            uses: actions/checkout@v5

          - name: setup .NET

            uses: actions/setup-dotnet@v4.3.1

            uses: actions/setup-dotnet@v5.0.0

          - name: Populate Databases.yaml

            shell: bash

            run: |

    @@ -84,9 +84,9 @@ jobs:
  
        runs-on: windows-latest

        steps:

          - name: Checkout code

            uses: actions/checkout@v4

            uses: actions/checkout@v5

          - name: setup .NET

            uses: actions/setup-dotnet@v4.3.1

            uses: actions/setup-dotnet@v5.0.0

          - name: Populate Databases.yaml

            shell: bash

            run: |

    @@ -152,15 +152,15 @@ jobs:
  
        runs-on: windows-latest

        steps:

          - name: Checkout code

            uses: actions/checkout@v4

            uses: actions/checkout@v5

          - name: setup .NET

            uses: actions/setup-dotnet@v4.3.1

            uses: actions/setup-dotnet@v5.0.0

          - name: Determine RDMP build version

            id: version

            shell: bash

            run: perl -ne "print \"rdmpversion=\$1\n\" if /AssemblyInformationalVersion\(\"([0-9a-z.-]+)\"\)/;" SharedAssemblyInfo.cs >> $GITHUB_OUTPUT

          - name: Setup .NET Core

            uses: actions/setup-dotnet@v4.3.1

            uses: actions/setup-dotnet@v5.0.0

            with:

              dotnet-version: 7.0.x

          - name: BundleSource

    @@ -204,7 +204,7 @@ jobs:
  
          - name: Sign & zip

            shell: bash

            run: |

              dotnet tool install --global AzureSignTool 

              dotnet tool install --global --version 6.0.1 AzureSignTool

              AzureSignTool sign -kvu "${{ secrets.AZURE_KEY_VAULT_URI }}" -kvi "${{ secrets.AZURE_CLIENT_ID }}" -kvt "${{ secrets.AZURE_TENANT_ID }}" -kvs "${{ secrets.AZURE_CLIENT_SECRET }}" -kvc ${{ secrets.AZURE_CERT_NAME }} -tr http://timestamp.digicert.com -v PublishWindows/rdmp.exe

              AzureSignTool sign -kvu "${{ secrets.AZURE_KEY_VAULT_URI }}" -kvi "${{ secrets.AZURE_CLIENT_ID }}" -kvt "${{ secrets.AZURE_TENANT_ID }}" -kvs "${{ secrets.AZURE_CLIENT_SECRET }}" -kvc ${{ secrets.AZURE_CERT_NAME }} -tr http://timestamp.digicert.com -v PublishWinForms/ResearchDataManagementPlatform.exe

              mkdir -p dist

    @@ -216,7 +216,7 @@ jobs:
  
          - name: Install Perl dependencies

            uses: shogo82148/actions-setup-perl@v1.34.0

            uses: shogo82148/actions-setup-perl@v1.36.0

            with:

              install-modules-with: cpanm

              install-modules: Archive::Zip Archive::Tar

    @@ -293,7 +293,7 @@ jobs:
  
              path: ${{ github.workspace }}/

              key: ${{ github.sha }}-your-cache-key-bundled

          - name: setup .NET

            uses: actions/setup-dotnet@v4.3.1

            uses: actions/setup-dotnet@v5.0.0

          - name: Determine RDMP build version

            id: version

            shell: bash

    @@ -307,4 +307,4 @@ jobs:
  
              file: dist/*

              tag: ${{ github.ref }}

              overwrite: true

              file_glob: true 

              file_glob: true

.github/workflows/codeql.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -24,15 +24,15 @@ jobs:
  
        steps:

          - name: Checkout

            uses: actions/checkout@v4

            uses: actions/checkout@v5

          - name: setup .NET

            uses: actions/setup-dotnet@v4.3.1

            uses: actions/setup-dotnet@v5.0.0

            with:

              global-json-file: global.json

              dotnet-version: '6.0.x'

          - name: Initialize CodeQL

            uses: github/codeql-action/init@v3

            uses: github/codeql-action/init@v4

            with:

              languages: ${{ matrix.language }}

              queries: +security-and-quality

    @@ -49,13 +49,13 @@ jobs:
  
              sed -i 's/"language": ""/"language": "en-US"/' sarif-results/scs.sarif

          - name: Perform CodeQL Analysis

            uses: github/codeql-action/analyze@v3

            uses: github/codeql-action/analyze@v4

            with:

              category: "/language:${{ matrix.language }}"

              upload: False

              output: sarif-results

          - name: Upload SARIF

            uses: github/codeql-action/upload-sarif@v3

            uses: github/codeql-action/upload-sarif@v4

            with:

              sarif_file: sarif-results

.github/workflows/docker.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -13,11 +13,11 @@ jobs:
  
          packages: write

        steps:

          - uses: actions/checkout@v4

          - uses: actions/checkout@v5

            with:

              submodules: true

          - name: Setup .NET Core

            uses: actions/setup-dotnet@v4.3.1

            uses: actions/setup-dotnet@v5.0.0

            with:

              dotnet-version: 6.0.x

          - name: Cache Nuget

.github/workflows/links.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -17,7 +17,7 @@ jobs: @@
         runs-on: ubuntu-latest
         steps:
           - name: Checkout
-            uses: actions/checkout@v4
+            uses: actions/checkout@v5
             with:
               submodules: recursive
           - name: Run link check
@@ Expand Down @@

Application/ResearchDataManagementPlatform/RDMPMainForm.cs

-Original file line number
+Diff line change
@@ Expand Up / @@ -105,6 +105,7 @@ private void RDMPMainForm_Load(object sender, EventArgs e) @@
             _rdmpTopMenuStrip1.InjectButton(exceptionCounter);
             _windowManager = new WindowManager(_theme, this, _refreshBus, dockPanel1, RepositoryLocator, exceptionCounter);
             SetItemActivator(_windowManager.ActivateItems);
             _rdmpTopMenuStrip1.SetWindowManager(_windowManager);
@@ Expand Down @@

Application/ResearchDataManagementPlatform/WindowManagement/ActivateItems.cs

-Original file line number
+Diff line change
@@ Expand Up / @@ -55,6 +55,7 @@ @@
     using Rdmp.UI.Refreshing;
     using Rdmp.UI.Rules;
     using Rdmp.UI.SimpleDialogs;
+    using Rdmp.UI.SimpleDialogs.Cohorts;
     using Rdmp.UI.SimpleDialogs.ForwardEngineering;
     using Rdmp.UI.SingleControlForms;
     using Rdmp.UI.SubComponents;
@@ Expand Down Expand Up @@
         public override IEnumerable<Type> GetIgnoredCommands()
         {
             yield return typeof(ExecuteCommandRefreshObject);
-            yield return typeof(ExecuteCommandChangeExtractability);
             yield return typeof(ExecuteCommandOpenInExplorer);
             yield return typeof(ExecuteCommandDeletePlugin);
             yield return typeof(ExecuteCommandCreateNewFileBasedProcessTask);
@@ Expand Down Expand Up @@
             return ui.ShowDialog() == DialogResult.OK ? ui.Result : null;
         }
+        public override IProject CohortCommitProjectSelect(IProject currentProject,Project[] projects)
+        {
+            var ui = new CohortCommitProjectSelectionUI(this, currentProject, projects);
+            return ui.ShowDialog() == DialogResult.OK ? ui.Result : null; ;
+        }
         public override CohortHoldoutLookupRequest GetCohortHoldoutLookupRequest(ExternalCohortTable externalCohortTable,
             IProject project, CohortIdentificationConfiguration cic)
         {
@@ Expand Down @@

CHANGELOG.md

-Original file line number
+Diff line change
@@ Expand Up @@
     The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
     and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+    ## [9.1.1] - 2025-12-02
+    - Allow Atlassian service workers to write to Confluence from RDMP
+    ## [9.1.0] - 2025-11-24
+    - Fix bug with duplicate searchables
+    - Improve UI for committing cohorts across projects
+    - Add the ability to Template Cohort Identification Configurations (see [Documentation\cohorts\CohortIdentificationConfigurationTemplates.md])
+    - Require all deletes to enter a commit message when using the commit system
+    - Introduce ability to view Catalogues in a flat view
+    - Fix bug where Internal catalogues were still able to be added to a CIC
+    - Add ability to use archive trigger when re-releasing to a database
+    ## [9.0.3] - 2025-11-03
+    - Improve checking for default pipelines
+    - Improve confluence catalogue metadata extraction
+    - Improve project specific catalogue error handling
+    ## [9.0.2] - 2025-10-21
+    - Add ability to view logs as a flat list. Default settings can be updated via user settings
+    - allow the use of Catalogue CIC filters in corresponding Extractions
+    - Add Data Load component to allow SQL to be executed on an external database server
+    - Improve Extraction Log Viewer Filter
+    - Add Dataset Variable document to extractions
+    - Update extractability to only allow non-internal projects to be extracted
+    - Remove functionality to mark individual catalogues as extractable/not extractable. Favor use of marking as internal.
+    - Move RSA key checks to fix bug where bad RSA keys were causing issues at launch
+    - Add cli command to export catalogue metadata to a Atlassian Confluence space
+    - Allow the use of network share RSA keys
+    ## [9.0.1] - 2025-07-31
+    - Improve Right-click performance
+    - Fix bug where orphaned filters were causing issues with cohort configurations
     ## [9.0.0] - 2025-07-23
     - Add ability to use Extraction Configuration ID in the naming scheme of the extract to database Pipeline component
     - Update Cohort Versioning Icons and Interface
@@ Expand Down @@

Directory.Packages.props

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,49 +1,49 @@
  
    <Project>

      <ItemGroup>

        <PackageVersion Include="AWSSDK.S3" Version="4.0.6" />

        <PackageVersion Include="AWSSDK.SecurityToken" Version="4.0.1.5" />

        <PackageVersion Include="AWSSDK.SSO" Version="4.0.0.14" />

        <PackageVersion Include="AWSSDK.SSOOIDC" Version="4.0.0.14" />

        <PackageVersion Include="AWSSDK.S3" Version="4.0.13.1" />

        <PackageVersion Include="AWSSDK.SecurityToken" Version="4.0.5.1" />

        <PackageVersion Include="AWSSDK.SSO" Version="4.0.2.5" />

        <PackageVersion Include="AWSSDK.SSOOIDC" Version="4.0.3.6" />

        <PackageVersion Include="CommandLineParser" Version="2.9.1" />

        <PackageVersion Include="Equ" Version="2.3.0" />

        <PackageVersion Include="ExcelNumberFormat" Version="1.1.0" />

        <PackageVersion Include="FluentFTP" Version="53.0.1" />

        <PackageVersion Include="FluentFTP" Version="53.0.2" />

        <PackageVersion Include="HIC.SynthEHR" Version="2.0.1" />

        <PackageVersion Include="HIC.FAnsiSql" Version="3.2.7" />

        <PackageVersion Include="LibArchive.Net" Version="0.1.5" />

        <PackageVersion Include="Microsoft.Bcl.AsyncInterfaces" Version="9.0.7" />

        <PackageVersion Include="Microsoft.Bcl.AsyncInterfaces" Version="9.0.10" />

        <PackageVersion Include="Microsoft.Net.Sdk.Compilers.Toolset" Version="9.0.300" />

        <PackageVersion Include="Microsoft.SourceLink.GitHub" Version="8.0.0">

          <PrivateAssets>all</PrivateAssets>

          <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>

        </PackageVersion>

        <PackageVersion Include="Minio" Version="6.0.5" />

        <PackageVersion Include="MongoDB.Driver" Version="3.4.1" />

        <PackageVersion Include="Newtonsoft.Json" Version="13.0.3" />

        <PackageVersion Include="NPOI" Version="2.7.4" />

        <PackageVersion Include="NLog" Version="6.0.2" />

        <PackageVersion Include="SixLabors.ImageSharp" Version="3.1.10" />

        <PackageVersion Include="SixLabors.ImageSharp.Drawing" Version="2.1.6" />

        <PackageVersion Include="Spectre.Console" Version="0.50.0" />

        <PackageVersion Include="SSH.NET" Version="2025.0.0" />

        <PackageVersion Include="Minio" Version="7.0.0" />

        <PackageVersion Include="MongoDB.Driver" Version="3.5.1" />

        <PackageVersion Include="Newtonsoft.Json" Version="13.0.4" />

        <PackageVersion Include="NPOI" Version="2.7.5" />

        <PackageVersion Include="NLog" Version="6.0.6" />

        <PackageVersion Include="SixLabors.ImageSharp" Version="3.1.12" />

        <PackageVersion Include="SixLabors.ImageSharp.Drawing" Version="2.1.7" />

        <PackageVersion Include="Spectre.Console" Version="0.54.0" />

        <PackageVersion Include="SSH.NET" Version="2025.1.0" />

        <PackageVersion Include="System.Private.Uri" Version="4.3.2" />

        <PackageVersion Include="Terminal.Gui" Version="1.19.0" />

        <PackageVersion Include="YamlDotNet" Version="16.3.0" />

        <PackageVersion Include="ConsoleControl" Version="1.3.0" />

        <PackageVersion Include="Autoupdater.NET.Official" Version="1.9.2" />

        <PackageVersion Include="DockPanelSuite.ThemeVS2015" Version="3.1.1" />

        <PackageVersion Include="System.Threading.ThreadPool" Version="4.3.0" />

        <PackageVersion Include="WeCantSpell.Hunspell" Version="6.0.0" />

        <PackageVersion Include="NUnit" Version="4.3.2" />

        <PackageVersion Include="WeCantSpell.Hunspell" Version="6.0.3" />

        <PackageVersion Include="NUnit" Version="4.4.0" />

        <PackageVersion Include="ObjectListView.Repack.NET6Plus" Version="2.9.5" />

        <PackageVersion Include="Scintilla.NET" Version="5.3.2.9" />

        <PackageVersion Include="System.Resources.Extensions" Version="9.0.7" />

        <PackageVersion Include="System.Resources.Extensions" Version="9.0.10" />

        <PackageVersion Include="VPKSoft.ScintillaLexers.NET" Version="1.1.16" />

        <PackageVersion Include="HIC.System.Windows.Forms.DataVisualization" Version="1.0.1" />

        <PackageVersion Include="NUnit3TestAdapter" Version="5.0.0" />

        <PackageVersion Include="NUnit.Analyzers" Version="4.9.2" />

        <PackageVersion Include="NUnit3TestAdapter" Version="5.2.0" />

        <PackageVersion Include="NUnit.Analyzers" Version="4.11.2" />

        <PackageVersion Include="NSubstitute" Version="5.3.0" />

        <PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />

        <PackageVersion Include="Microsoft.NET.Test.Sdk" Version="18.0.1" />

        <PackageVersion Include="coverlet.collector" Version="6.0.4" />

      </ItemGroup>

    </Project>

Documentation/CodeTutorials/CustomMetadataSubstitutions.md

-Original file line number
+Diff line change
@@ Expand Up @@
     |$Geographical_coverage|User provided |
     |$Granularity|User provided |
     |$ID|RDMP allocated number uniquely identifying this object instance in this RDMP installation|
-    |$IsColdStorageDataset|True or False flag, configurable in RDMP|
     |$IsDeprecated|True or False flag, configurable in RDMP|
     |$IsInternalDataset|True or False flag, configurable in RDMP|
     |$Last_revision_date|User provided|
@@ Expand Down @@

Documentation/Cohorts/CohortIdentificationConfigurationTemplates.md

-Original file line number
+Diff line change
@@ -0,0 +1,38 @@
+    ## Cohort Identification Configuration Templates
+    Cohort Identification Configurations (CICs) can be turned into Templates for future re-use.
+    Templates offer certain functionality that differs from standard CICs. They:
+    * Cannot be edited
+    * Cannot be used to commit a cohort
+    * Are stored in a separate Templates section of the Cohort Builder
+    Templates are designed to be a base to build your CIC from, saving you from having to copy, add, and edit boilerplate CIC configuration each and every time.
+    ### Creating a Template
+    To create a new CIC Template:
+. Begin by creating a new CIC (or use an existing CIC).
+.Update your CIC to create the configuration you wish to template.
+. Once ready to turn this CIC into a template, right click on the CIC in the cohort builder tree.
+. Select the option "Create Cohort Identification Template".
+. If your CIC is associated with a [Project], it will ask you some questions about how you would like to associate the template
+. Congratulations, You've just made a CIC template
+    ### Using a Template
+    To use a CIC Template:
+. Right click on an existing CIC Template
+. Select the "Use Template Cohort Identification Configuration" option
+. If your template CIC is associated with a [Project], it will ask you some questions about how you would like to associate the new CIC
+. Congratulations, You've just made a new CIC from a template
+    ### Associating with a [Project]
+    Just like associating a CIC with a [Project], CIC Templates can also be associated with [Project]s.
+    While these templates can still be used outside the associated [Project], it does allow you to see the template from within the [Project] view.
+    This may be helpful to keep track of what templates are typically used with each [Project]
+    [Project]: ../CodeTutorials/Glossary.md#Project

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Datasets] Task/rdmp 33 dataset integration interface #2182

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

[Datasets] Task/rdmp 33 dataset integration interface #2182

Are you sure you want to change the base?

Uh oh!

[Datasets] Task/rdmp 33 dataset integration interface #2182

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!