Skip to content

Conversation

@SebastianClaesson
Copy link
Contributor

Description

Fixes #1368

Adding 31 Azure DevOps security tests.
Each of the tests is added to the new folder azdo in maester\public\maester

All of the tests have a markdown file, with :

  • Rationale
  • Remediation action
  • Related links (Official links / blog posts describing the security issue)

Contribution Checklist

Before submitting this PR, please confirm you have completed the following:

  • 📖 Read the guidelines for contributing to this repository.
  • 🧪 Ensure the build and unit tests pass by running /powershell/tests/pester.ps1 on your local system.

@SebastianClaesson SebastianClaesson requested review from a team as code owners December 11, 2025 08:06
@SamErde SamErde requested a review from Copilot January 21, 2026 11:19
@SamErde SamErde added enhancement New feature or request maester-test Related to a Maester test labels Jan 21, 2026
3. Select Policies, locate the Request Access policy and toggle it to off.
4. Provide the URL to your internal process for gaining access. Users see this URL in the error report when they try to access the organization or a project within the organization that they don't have permission to access.

**Results:**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the detailed 401 go to users in the organization and the 404 go to users not in the organization?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do agree!
Seems the experience has changed since;

MicrosoftDocs/azure-devops-docs@91c4410#diff-a28e8dd823f1493651d0c6322e6b2dd976bccab79e279de5aea876ce20272729R44

I'll update with the new information the article.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds 31 Azure DevOps security tests to the Maester project, providing comprehensive security assessments for Azure DevOps organizations. Each test includes both PowerShell implementation and markdown documentation describing the security rationale and remediation steps.

Changes:

  • Added 31 new Azure DevOps security test functions covering authentication, access control, pipeline security, and resource management
  • Created comprehensive markdown documentation for each test with remediation guidance
  • Updated the module manifest to export all new test functions
  • Added a test runner file that orchestrates all Azure DevOps security tests

Reviewed changes

Copilot reviewed 65 out of 65 changed files in this pull request and generated 28 comments.

Show a summary per file
File Description
tests/Maester/Azdo/Test-Azdo.Tests.ps1 Test runner that executes all 31 Azure DevOps security tests
tests/Maester/Azdo/README.md Overview documentation for Azure DevOps tests
powershell/public/maester/azdo/*.ps1 31 PowerShell functions implementing security checks
powershell/public/maester/azdo/*.md 31 markdown documentation files with rationale and remediation steps
powershell/Maester.psd1 Module manifest updated to export new functions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.


It "AZDO.1024: Disable Node 6 tasks. See https://learn.microsoft.com/en-us/azure/devops/release-notes/roadmap/2022/no-node-6-on-hosted-agents" -Tag "AZDO.1024" {

Test-AzdoOrganizationTaskRestrictionsDisableNode6Tasks | Should -Be $true -Because "With this enabled, pipelines will fail if they utilize a task with a Node 6 execution handler."
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationTaskRestrictionsDisableNode6Tasks (plural) but the function is named Test-AzdoOrganizationTaskRestrictionsDisableNode6Task (singular).

Copilot uses AI. Check for mistakes.

It "AZDO.1027: Disable showing Gravatar images for users outside of your enterprise. See https://learn.microsoft.com/en-us/azure/devops/repos/git/repository-settings?view=azure-devops&tabs=browser#gravatar-images" -Tag "AZDO.1027" {

Test-AzdoOrganizationRepositorySettingsGravatarImages | Should -Be $false -Because "Gravatar images should not be exposed outside of your enterprise."
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationRepositorySettingsGravatarImages (plural) but the function is named Test-AzdoOrganizationRepositorySettingsGravatarImage (singular).

Copilot uses AI. Check for mistakes.

It "AZDO.1012: Work Items Tags Limits. See https://learn.microsoft.com/en-us/azure/devops/organizations/settings/work/object-limits?view=azure-devops" -Tag "AZDO.1012" {

Test-AzdoResourceUsageWorkItemTags | Should -Be $true -Because "Azure DevOps supports up to 150,000 tag definitions per organization or collection."
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoResourceUsageWorkItemTags (plural) but the function is named Test-AzdoResourceUsageWorkItemTag (singular).

Copilot uses AI. Check for mistakes.

It "AZDO.1016: Limit job authorization scope to current project for non-release pipelines. See https://learn.microsoft.com/en-us/azure/devops/pipelines/process/access-tokens?view=azure-devops&tabs=yaml#job-authorization-scope" -Tag "AZDO.1016" {

Test-AzdoOrganizationLimitJobAuthorizationScopeNonReleasePipelines | Should -Be $true -Because "With this option enabled, you can reduce the scope of access for all classic release pipelines to the current project."
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationLimitJobAuthorizationScopeNonReleasePipelines (plural) but the function is named Test-AzdoOrganizationLimitJobAuthorizationScopeNonReleasePipeline (singular).

Copilot uses AI. Check for mistakes.

It "AZDO.1023: Disable Marketplace tasks. See https://learn.microsoft.com/en-us/azure/devops/pipelines/security/overview?view=azure-devops#prevent-malicious-code-execution" -Tag "AZDO.1023" {

Test-AzdoOrganizationTaskRestrictionsDisableMarketplaceTasks | Should -Be $false -Because "Disable the ability to install and run tasks from the Marketplace, which gives you greater control over the code that executes in a pipeline."
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationTaskRestrictionsDisableMarketplaceTasks (plural) but the function is named Test-AzdoOrganizationTaskRestrictionsDisableMarketplaceTask (singular).

Copilot uses AI. Check for mistakes.

It "AZDO.1026: Enable automatic enrollment to Advanced Security for Azure DevOps. See https://learn.microsoft.com/en-us/azure/devops/repos/security/configure-github-advanced-security-features?view=azure-devops&tabs=yaml#organization-level-onboarding" -Tag "AZDO.1026" {

Test-AzdoOrganizationAutomaticEnrollmentAdvancedSecurityNewProjects | Should -Be $true -Because "Enable automatic enrollment for new git repositories to use GitHub Advanced Security for Azure DevOps. It adds GitHub Advanced Security's suite of security features to Azure Repos."
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function name called in the test does not match the actual function name defined in the PowerShell file. The test calls Test-AzdoOrganizationAutomaticEnrollmentAdvancedSecurityNewProjects (plural) but the function is named Test-AzdoOrganizationAutomaticEnrollmentAdvancedSecurityNewProject (singular).

Copilot uses AI. Check for mistakes.
Corrected typos and formatting in the documentation.
Removed unnecessary commas and spaces. Added line breaks for MD linting.
Corrected grammatical errors and added punctuation for clarity.
Updated description to clarify the function's purpose and improve readability.
Corrected grammatical errors and improved clarity in remediation instructions.
Clarified the purpose of the Azure DevOps tests and updated the reference link.
…thorizationScopeNonReleasePipeline.ps1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@SamErde
Copy link
Contributor

SamErde commented Jan 21, 2026

This is an awesome addition, @SebastianClaesson! Please review these suggestions from the initial review. I have gone through a number of the files and made some minor edits to spelling, punctuation, etc; Copilot has found a few other potential issues.

I'm also curious if @maester365/core-tests has any thoughts on standardizing the location for added tests. Would the best location for these be in maester/tests/Maester/Azdo or in maester/tests/Azdo?

(Super minor nit-pick: I'm not sure how I feel about the casing of 'Azdo' vs 'azdo' vs potentially just using 'AzureDevOps'.) 🤓

@SebastianClaesson
Copy link
Contributor Author

This is an awesome addition, @SebastianClaesson! Please review these suggestions from the initial review. I have gone through a number of the files and made some minor edits to spelling, punctuation, etc; Copilot has found a few other potential issues.

I'm also curious if @maester365/core-tests has any thoughts on standardizing the location for added tests. Would the best location for these be in maester/tests/Maester/Azdo or in maester/tests/Azdo?

(Super minor nit-pick: I'm not sure how I feel about the casing of 'Azdo' vs 'azdo' vs potentially just using 'AzureDevOps'.) 🤓

I agree with the 'AzureDevOps' naming. I think that makes sense, as Azdo might be an abbreviation for something else for some!

Thank you so much, I really apprieciate you taking the time for feedback and ensuring high quality!

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 65 out of 65 changed files in this pull request and generated 13 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +29 to +36
$ApplicationPolicies = Get-ADOPSOrganizationPolicy -PolicyCategory 'ApplicationConnection'
$Policy = $ApplicationPolicies.policy | where-object -property name -eq 'Policy.DisallowOAuthAuthentication'
$result = $Policy.effectiveValue
if ($result) {
$resultMarkdown = "Your tenant have not restricted Azure DevOps OAuth apps to access resources in your organization through OAuth."
} else {
$resultMarkdown = "Well done. Your tenant has restricted Azure DevOps OAuth apps to access resources in your organization through OAuth."
}
Copy link

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function currently returns the raw value of the Policy.DisallowOAuthAuthentication policy, so when the policy is enabled (OAuth apps are disallowed) $result is $true, but the success message is written in the else branch and the Pester test expects the secure configuration to yield $false. That makes the boolean result and messages inconsistent with the policy name and the test; you likely want to invert the effective value for the return value and align the "well done"/"not restricted" messages with the actual secure vs insecure states.

Copilot uses AI. Check for mistakes.
Comment on lines +5 to +10
#### Remediation action:
Disable the policy to stops these requests and notifications.
1. Sign in to your organization
2. Choose Organization settings.
3. Select Policies, locate the Request Access policy and toggle it to off.
4. Provide the URL to your internal process for gaining access. Users see this URL in the error report when they try to access the organization or a project within the organization that they don't have permission to access.
Copy link

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The remediation steps here describe disabling the Request Access policy and configuring an internal access URL, which is unrelated to third-party application access via OAuth and duplicates the guidance from the Request Access test. This will send users to the wrong setting; please update the remediation section to reference the correct Azure DevOps policy for third‑party OAuth apps and remove the Request Access–specific steps.

Copilot uses AI. Check for mistakes.
Comment on lines +28 to +35
$ApplicationPolicies = Get-ADOPSOrganizationPolicy -PolicyCategory 'ApplicationConnection'
$Policy = $ApplicationPolicies.policy | where-object -property name -eq 'Policy.DisallowSecureShell'
$result = $Policy.effectiveValue
if ($result) {
$resultMarkdown = "Your tenant allows developers to connect to your Git repos through SSH on macOS, Linux, or Windows to connect with Azure DevOps"
} else {
$resultMarkdown = "Well done. Your tenant do not allow developers to connect to your Git repos through SSH on macOS, Linux, or Windows to connect with Azure DevOps"
}
Copy link

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function uses the Policy.DisallowSecureShell effective value directly, but the messages are inverted: when the policy is true (SSH disallowed), the text says "allows developers to connect", and when it's false it says "do not allow developers to connect". In addition, the Pester test asserts Should -Be $false for the secure configuration, so you should either invert the policy value for the return value or fix the messages/tests so that the returned boolean and narrative consistently reflect whether SSH access is permitted.

Copilot uses AI. Check for mistakes.
Comment on lines +33 to +42
$result = $Policy.effectiveValue
if ($result) {
$resultMarkdown = "External user(s) can be added to the organization to which they were invited and has immediate access. A guest user can add other guest users to the organization after being granted the Guest Inviter role in Microsoft Entra ID."
}
else {
$resultMarkdown = "Well done. External users should not be allowed access to your Azure DevOps organization"
}



Copy link

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Policy.DisallowAadGuestUserAccess effective value is returned directly here, but the message and test expectations appear inverted: when $result is $true you describe guests being able to be added, and when it's $false you print "Well done. External users should not be allowed access", while the Pester test asserts Should -Be $false for the secure configuration. Please make the boolean return and messages consistently indicate whether guest access is allowed (likely by inverting the policy value and updating the "Well done" branch) so that the test result matches the actual security posture.

Suggested change
$result = $Policy.effectiveValue
if ($result) {
$resultMarkdown = "External user(s) can be added to the organization to which they were invited and has immediate access. A guest user can add other guest users to the organization after being granted the Guest Inviter role in Microsoft Entra ID."
}
else {
$resultMarkdown = "Well done. External users should not be allowed access to your Azure DevOps organization"
}
# Interpret result as "guest access is allowed" for consistency with messages and tests.
$result = -not $Policy.effectiveValue
if ($result) {
$resultMarkdown = "External user(s) can be added to the organization to which they were invited and has immediate access. A guest user can add other guest users to the organization after being granted the Guest Inviter role in Microsoft Entra ID."
}
else {
$resultMarkdown = "Well done. External users are not allowed access to your Azure DevOps organization."
}

Copilot uses AI. Check for mistakes.
Comment on lines +28 to +36
$result = (Get-ADOPSOrganizationPipelineSettings).disableStageChooser

if ($result) {
$resultMarkdown = "Well done. Users will not be able to select stages to skip from the Queue Pipeline panel."
$result = $false
}
else {
$resultMarkdown = "Users are able to select stages to skip from the Queue Pipeline panel."
$result = $true
Copy link

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function sets $result = (Get-ADOPSOrganizationPipelineSettings).disableStageChooser, then immediately overwrites $result to $false in the "disabled" branch and $true in the "enabled" branch. Although this happens to satisfy the current test expectation (Should -Be $false when the stage chooser is disabled), it makes $result no longer reflect the underlying setting and is confusing to maintain; consider keeping $result as the raw setting and returning its negation (or using a separate variable) instead of mutating it.

Copilot uses AI. Check for mistakes.
Comment on lines +8 to +9
- Increate Artifacts storage limit
- [Set up billing for your organization.](https://learn.microsoft.com/en-us/azure/devops/organizations/billing/set-up-billing-for-your-organization-vs?view=azure-devops#set-up-billing)
Copy link

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in the remediation bullet: "Increate Artifacts storage limit" should be "Increase Artifacts storage limit".

Copilot uses AI. Check for mistakes.

#### Related links

* [Learn - Removal of RFVC in new projects](https://learn.microsoft.com/en-us/azure/devops/release-notes/roadmap/2024/no-tfvc-in-new-projects)
Copy link

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in the related link label: "Removal of RFVC in new projects" should read "Removal of TFVC in new projects" to match the technology name used elsewhere in this document.

Copilot uses AI. Check for mistakes.
@@ -0,0 +1,18 @@
Connecting to Azure DevOps using SSH should be disabled.

Rationale: Oauth is the prefered and most secure authentication method.
Copy link

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in the rationale sentence: "Oauth is the prefered and most secure authentication method." should be "OAuth is the preferred and most secure authentication method."

Copilot uses AI. Check for mistakes.
Comment on lines +13 to +17
It "AZDO.1002: Log Audit Events. See https://aka.ms/log-audit-events" -Tag "AZDO.1002" {

Test-AzdoLogAuditEvents | Should -Be $true -Because "Auditing should be enabled for Azure DevOps"
}

Copy link

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests reference several Azure DevOps test functions with pluralized names (for example Test-AzdoLogAuditEvents, Test-AzdoPublicProjects, Test-AzdoAuditStreams, Test-AzdoResourceUsageProjects, etc.), but the functions defined in powershell/public/maester/azuredevops and exported in Maester.psd1 use the singular form (e.g. Test-AzdoLogAuditEvent, Test-AzdoPublicProject, Test-AzdoAuditStream, Test-AzdoResourceUsageProject). As written, these Pester tests will fail with "command not found" errors; please align the test names with the actual function names (or vice versa) and keep the manifest export list consistent.

Copilot uses AI. Check for mistakes.
…itJobAuthorizationScopeReleasePipeline.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@merill
Copy link
Contributor

merill commented Feb 4, 2026

@SebastianClaesson this is top notch work! Love it.

I would suggest two additions.

  1. Add doc - Update the installation doc to include the optinal module to install and the command for connecting website/docs/installation.md . See https://maester.dev/docs/installation#optional-modules-and-permissions
  2. Add a blog post that outlines that calls out the new tests to website/blog See https://maester.dev/blog

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request maester-test Related to a Maester test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🙏 Azure DevOps tests

3 participants