-
Notifications
You must be signed in to change notification settings - Fork 221
Feature/azdo #1371
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Feature/azdo #1371
Conversation
…aester into feature/azdo
| 3. Select Policies, locate the Request Access policy and toggle it to off. | ||
| 4. Provide the URL to your internal process for gaining access. Users see this URL in the error report when they try to access the organization or a project within the organization that they don't have permission to access. | ||
|
|
||
| **Results:** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't the detailed 401 go to users in the organization and the 404 go to users not in the organization?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do agree!
Seems the experience has changed since;
I'll update with the new information the article.
powershell/public/maester/azdo/Test-AzdoAllowRequestAccessToken.md
Outdated
Show resolved
Hide resolved
powershell/public/maester/azdo/Test-AzdoAllowTeamAdminsInvitationsAccessToken.md
Outdated
Show resolved
Hide resolved
powershell/public/maester/azdo/Test-AzdoAllowTeamAdminsInvitationsAccessToken.md
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This pull request adds 31 Azure DevOps security tests to the Maester project, providing comprehensive security assessments for Azure DevOps organizations. Each test includes both PowerShell implementation and markdown documentation describing the security rationale and remediation steps.
Changes:
- Added 31 new Azure DevOps security test functions covering authentication, access control, pipeline security, and resource management
- Created comprehensive markdown documentation for each test with remediation guidance
- Updated the module manifest to export all new test functions
- Added a test runner file that orchestrates all Azure DevOps security tests
Reviewed changes
Copilot reviewed 65 out of 65 changed files in this pull request and generated 28 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/Maester/Azdo/Test-Azdo.Tests.ps1 | Test runner that executes all 31 Azure DevOps security tests |
| tests/Maester/Azdo/README.md | Overview documentation for Azure DevOps tests |
| powershell/public/maester/azdo/*.ps1 | 31 PowerShell functions implementing security checks |
| powershell/public/maester/azdo/*.md | 31 markdown documentation files with rationale and remediation steps |
| powershell/Maester.psd1 | Module manifest updated to export new functions |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...ll/public/maester/azdo/Test-AzdoOrganizationLimitJobAuthorizationScopeNonReleasePipeline.ps1
Outdated
Show resolved
Hide resolved
|
|
||
| It "AZDO.1024: Disable Node 6 tasks. See https://learn.microsoft.com/en-us/azure/devops/release-notes/roadmap/2022/no-node-6-on-hosted-agents" -Tag "AZDO.1024" { | ||
|
|
||
| Test-AzdoOrganizationTaskRestrictionsDisableNode6Tasks | Should -Be $true -Because "With this enabled, pipelines will fail if they utilize a task with a Node 6 execution handler." |
Copilot
AI
Jan 21, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationTaskRestrictionsDisableNode6Tasks (plural) but the function is named Test-AzdoOrganizationTaskRestrictionsDisableNode6Task (singular).
|
|
||
| It "AZDO.1027: Disable showing Gravatar images for users outside of your enterprise. See https://learn.microsoft.com/en-us/azure/devops/repos/git/repository-settings?view=azure-devops&tabs=browser#gravatar-images" -Tag "AZDO.1027" { | ||
|
|
||
| Test-AzdoOrganizationRepositorySettingsGravatarImages | Should -Be $false -Because "Gravatar images should not be exposed outside of your enterprise." |
Copilot
AI
Jan 21, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationRepositorySettingsGravatarImages (plural) but the function is named Test-AzdoOrganizationRepositorySettingsGravatarImage (singular).
|
|
||
| It "AZDO.1012: Work Items Tags Limits. See https://learn.microsoft.com/en-us/azure/devops/organizations/settings/work/object-limits?view=azure-devops" -Tag "AZDO.1012" { | ||
|
|
||
| Test-AzdoResourceUsageWorkItemTags | Should -Be $true -Because "Azure DevOps supports up to 150,000 tag definitions per organization or collection." |
Copilot
AI
Jan 21, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoResourceUsageWorkItemTags (plural) but the function is named Test-AzdoResourceUsageWorkItemTag (singular).
|
|
||
| It "AZDO.1016: Limit job authorization scope to current project for non-release pipelines. See https://learn.microsoft.com/en-us/azure/devops/pipelines/process/access-tokens?view=azure-devops&tabs=yaml#job-authorization-scope" -Tag "AZDO.1016" { | ||
|
|
||
| Test-AzdoOrganizationLimitJobAuthorizationScopeNonReleasePipelines | Should -Be $true -Because "With this option enabled, you can reduce the scope of access for all classic release pipelines to the current project." |
Copilot
AI
Jan 21, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationLimitJobAuthorizationScopeNonReleasePipelines (plural) but the function is named Test-AzdoOrganizationLimitJobAuthorizationScopeNonReleasePipeline (singular).
|
|
||
| It "AZDO.1023: Disable Marketplace tasks. See https://learn.microsoft.com/en-us/azure/devops/pipelines/security/overview?view=azure-devops#prevent-malicious-code-execution" -Tag "AZDO.1023" { | ||
|
|
||
| Test-AzdoOrganizationTaskRestrictionsDisableMarketplaceTasks | Should -Be $false -Because "Disable the ability to install and run tasks from the Marketplace, which gives you greater control over the code that executes in a pipeline." |
Copilot
AI
Jan 21, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationTaskRestrictionsDisableMarketplaceTasks (plural) but the function is named Test-AzdoOrganizationTaskRestrictionsDisableMarketplaceTask (singular).
|
|
||
| It "AZDO.1026: Enable automatic enrollment to Advanced Security for Azure DevOps. See https://learn.microsoft.com/en-us/azure/devops/repos/security/configure-github-advanced-security-features?view=azure-devops&tabs=yaml#organization-level-onboarding" -Tag "AZDO.1026" { | ||
|
|
||
| Test-AzdoOrganizationAutomaticEnrollmentAdvancedSecurityNewProjects | Should -Be $true -Because "Enable automatic enrollment for new git repositories to use GitHub Advanced Security for Azure DevOps. It adds GitHub Advanced Security's suite of security features to Azure Repos." |
Copilot
AI
Jan 21, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationAutomaticEnrollmentAdvancedSecurityNewProjects (plural) but the function is named Test-AzdoOrganizationAutomaticEnrollmentAdvancedSecurityNewProject (singular).
powershell/public/maester/azuredevops/Test-AzdoOrganizationStageChooser.md
Outdated
Show resolved
Hide resolved
powershell/public/maester/azuredevops/Test-AzdoSSHAuthentication.md
Outdated
Show resolved
Hide resolved
Corrected typos and formatting in the documentation.
Removed unnecessary commas and spaces. Added line breaks for MD linting.
Corrected grammatical errors and added punctuation for clarity.
Updated description to clarify the function's purpose and improve readability.
Corrected grammatical errors and improved clarity in remediation instructions.
Clarified the purpose of the Azure DevOps tests and updated the reference link.
…thorizationScopeNonReleasePipeline.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
This is an awesome addition, @SebastianClaesson! Please review these suggestions from the initial review. I have gone through a number of the files and made some minor edits to spelling, punctuation, etc; Copilot has found a few other potential issues. I'm also curious if @maester365/core-tests has any thoughts on standardizing the location for added tests. Would the best location for these be in (Super minor nit-pick: I'm not sure how I feel about the casing of 'Azdo' vs 'azdo' vs potentially just using 'AzureDevOps'.) 🤓 |
I agree with the 'AzureDevOps' naming. I think that makes sense, as Azdo might be an abbreviation for something else for some! Thank you so much, I really apprieciate you taking the time for feedback and ensuring high quality! |
…aester into feature/azdo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 65 out of 65 changed files in this pull request and generated 13 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| $ApplicationPolicies = Get-ADOPSOrganizationPolicy -PolicyCategory 'ApplicationConnection' | ||
| $Policy = $ApplicationPolicies.policy | where-object -property name -eq 'Policy.DisallowOAuthAuthentication' | ||
| $result = $Policy.effectiveValue | ||
| if ($result) { | ||
| $resultMarkdown = "Your tenant have not restricted Azure DevOps OAuth apps to access resources in your organization through OAuth." | ||
| } else { | ||
| $resultMarkdown = "Well done. Your tenant has restricted Azure DevOps OAuth apps to access resources in your organization through OAuth." | ||
| } |
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function currently returns the raw value of the Policy.DisallowOAuthAuthentication policy, so when the policy is enabled (OAuth apps are disallowed) $result is $true, but the success message is written in the else branch and the Pester test expects the secure configuration to yield $false. That makes the boolean result and messages inconsistent with the policy name and the test; you likely want to invert the effective value for the return value and align the "well done"/"not restricted" messages with the actual secure vs insecure states.
| #### Remediation action: | ||
| Disable the policy to stops these requests and notifications. | ||
| 1. Sign in to your organization | ||
| 2. Choose Organization settings. | ||
| 3. Select Policies, locate the Request Access policy and toggle it to off. | ||
| 4. Provide the URL to your internal process for gaining access. Users see this URL in the error report when they try to access the organization or a project within the organization that they don't have permission to access. |
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The remediation steps here describe disabling the Request Access policy and configuring an internal access URL, which is unrelated to third-party application access via OAuth and duplicates the guidance from the Request Access test. This will send users to the wrong setting; please update the remediation section to reference the correct Azure DevOps policy for third‑party OAuth apps and remove the Request Access–specific steps.
| $ApplicationPolicies = Get-ADOPSOrganizationPolicy -PolicyCategory 'ApplicationConnection' | ||
| $Policy = $ApplicationPolicies.policy | where-object -property name -eq 'Policy.DisallowSecureShell' | ||
| $result = $Policy.effectiveValue | ||
| if ($result) { | ||
| $resultMarkdown = "Your tenant allows developers to connect to your Git repos through SSH on macOS, Linux, or Windows to connect with Azure DevOps" | ||
| } else { | ||
| $resultMarkdown = "Well done. Your tenant do not allow developers to connect to your Git repos through SSH on macOS, Linux, or Windows to connect with Azure DevOps" | ||
| } |
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function uses the Policy.DisallowSecureShell effective value directly, but the messages are inverted: when the policy is true (SSH disallowed), the text says "allows developers to connect", and when it's false it says "do not allow developers to connect". In addition, the Pester test asserts Should -Be $false for the secure configuration, so you should either invert the policy value for the return value or fix the messages/tests so that the returned boolean and narrative consistently reflect whether SSH access is permitted.
| $result = $Policy.effectiveValue | ||
| if ($result) { | ||
| $resultMarkdown = "External user(s) can be added to the organization to which they were invited and has immediate access. A guest user can add other guest users to the organization after being granted the Guest Inviter role in Microsoft Entra ID." | ||
| } | ||
| else { | ||
| $resultMarkdown = "Well done. External users should not be allowed access to your Azure DevOps organization" | ||
| } | ||
|
|
||
|
|
||
|
|
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Policy.DisallowAadGuestUserAccess effective value is returned directly here, but the message and test expectations appear inverted: when $result is $true you describe guests being able to be added, and when it's $false you print "Well done. External users should not be allowed access", while the Pester test asserts Should -Be $false for the secure configuration. Please make the boolean return and messages consistently indicate whether guest access is allowed (likely by inverting the policy value and updating the "Well done" branch) so that the test result matches the actual security posture.
| $result = $Policy.effectiveValue | |
| if ($result) { | |
| $resultMarkdown = "External user(s) can be added to the organization to which they were invited and has immediate access. A guest user can add other guest users to the organization after being granted the Guest Inviter role in Microsoft Entra ID." | |
| } | |
| else { | |
| $resultMarkdown = "Well done. External users should not be allowed access to your Azure DevOps organization" | |
| } | |
| # Interpret result as "guest access is allowed" for consistency with messages and tests. | |
| $result = -not $Policy.effectiveValue | |
| if ($result) { | |
| $resultMarkdown = "External user(s) can be added to the organization to which they were invited and has immediate access. A guest user can add other guest users to the organization after being granted the Guest Inviter role in Microsoft Entra ID." | |
| } | |
| else { | |
| $resultMarkdown = "Well done. External users are not allowed access to your Azure DevOps organization." | |
| } |
| $result = (Get-ADOPSOrganizationPipelineSettings).disableStageChooser | ||
|
|
||
| if ($result) { | ||
| $resultMarkdown = "Well done. Users will not be able to select stages to skip from the Queue Pipeline panel." | ||
| $result = $false | ||
| } | ||
| else { | ||
| $resultMarkdown = "Users are able to select stages to skip from the Queue Pipeline panel." | ||
| $result = $true |
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function sets $result = (Get-ADOPSOrganizationPipelineSettings).disableStageChooser, then immediately overwrites $result to $false in the "disabled" branch and $true in the "enabled" branch. Although this happens to satisfy the current test expectation (Should -Be $false when the stage chooser is disabled), it makes $result no longer reflect the underlying setting and is confusing to maintain; consider keeping $result as the raw setting and returning its negation (or using a separate variable) instead of mutating it.
| - Increate Artifacts storage limit | ||
| - [Set up billing for your organization.](https://learn.microsoft.com/en-us/azure/devops/organizations/billing/set-up-billing-for-your-organization-vs?view=azure-devops#set-up-billing) |
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo in the remediation bullet: "Increate Artifacts storage limit" should be "Increase Artifacts storage limit".
|
|
||
| #### Related links | ||
|
|
||
| * [Learn - Removal of RFVC in new projects](https://learn.microsoft.com/en-us/azure/devops/release-notes/roadmap/2024/no-tfvc-in-new-projects) |
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo in the related link label: "Removal of RFVC in new projects" should read "Removal of TFVC in new projects" to match the technology name used elsewhere in this document.
| @@ -0,0 +1,18 @@ | |||
| Connecting to Azure DevOps using SSH should be disabled. | |||
|
|
|||
| Rationale: Oauth is the prefered and most secure authentication method. | |||
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo in the rationale sentence: "Oauth is the prefered and most secure authentication method." should be "OAuth is the preferred and most secure authentication method."
...public/maester/azuredevops/Test-AzdoOrganizationLimitJobAuthorizationScopeReleasePipeline.md
Outdated
Show resolved
Hide resolved
| It "AZDO.1002: Log Audit Events. See https://aka.ms/log-audit-events" -Tag "AZDO.1002" { | ||
|
|
||
| Test-AzdoLogAuditEvents | Should -Be $true -Because "Auditing should be enabled for Azure DevOps" | ||
| } | ||
|
|
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The tests reference several Azure DevOps test functions with pluralized names (for example Test-AzdoLogAuditEvents, Test-AzdoPublicProjects, Test-AzdoAuditStreams, Test-AzdoResourceUsageProjects, etc.), but the functions defined in powershell/public/maester/azuredevops and exported in Maester.psd1 use the singular form (e.g. Test-AzdoLogAuditEvent, Test-AzdoPublicProject, Test-AzdoAuditStream, Test-AzdoResourceUsageProject). As written, these Pester tests will fail with "command not found" errors; please align the test names with the actual function names (or vice versa) and keep the manifest export list consistent.
…itJobAuthorizationScopeReleasePipeline.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
@SebastianClaesson this is top notch work! Love it. I would suggest two additions.
|
Description
Fixes #1368
Adding 31 Azure DevOps security tests.
Each of the tests is added to the new folder azdo in maester\public\maester
All of the tests have a markdown file, with :
Contribution Checklist
Before submitting this PR, please confirm you have completed the following:
/powershell/tests/pester.ps1on your local system.