diff --git a/docs/development-guidelines/definition-of-done.md b/docs/development-guidelines/definition-of-done.md index 79b1fd2..76be1e3 100644 --- a/docs/development-guidelines/definition-of-done.md +++ b/docs/development-guidelines/definition-of-done.md @@ -14,9 +14,6 @@ a functional application. ## Acceptance criteria for gallery acceptance -> [WARNING] -> This is a draft - The following checklist must be complete before a template is published # Repository Management @@ -36,6 +33,7 @@ The following checklist must be complete before a template is published - [ ] DevContainer (/.devcontainer folder where applicable) configuration is in place - [ ] Infrastructure as code is in place (`/infra` folder where applicable, manifest files or code generators in the case of `Aspire` and similar ) - [ ] Azure services configuration (/azure.yml file) is in place +- [ ] end-to-end testing with Playwright is in place, as specified in [these guidelines](./e2e-testing.md) ## Functional requirements @@ -51,17 +49,25 @@ In the absense of e2e tests, we kindly ask you to make sure that ## Security requirements -- [ ] Microsoft Managed Identity is implemented to authenticate to Azure AI Services +- [ ] Microsoft Managed Identity is used for the following services -When a service selected doesn't support Managed Identity, the corresponding issue must have been reported and the security considerations section in the readme, should clearly explain the alternatives. +- Azure AI +- Azure APIM +- Azure App Service +- Azure Container Apps +- Azure Event Grid +- Azure Kubernetes Service +- Azure Search +- Azure SQL Server -- Azure Key Vault is a preferred alternative +considered our Hero Services. -### The following items are not strictly enforced but may prevent the template from being added to the gallery +When a service selected doesn't support Managed Identity, the corresponding issue must have been reported and the security considerations section in the readme, should clearly explain the alternatives. -#### Security requirements for production only +- Azure Key Vault is a preferred alternative -- [ ] Microsoft Managed Identity is used for all services and API endpoints +> [!IMPORTANT] +> Please note that this list may be regularly updated #### Code quality and integrity diff --git a/docs/development-guidelines/e2e-testing.md b/docs/development-guidelines/e2e-testing.md new file mode 100644 index 0000000..7ef2490 --- /dev/null +++ b/docs/development-guidelines/e2e-testing.md @@ -0,0 +1,222 @@ +# e2e testing to validate application and API endpoints + +While the validation pipeline automates the process of testing standards compliance for documentation, including the README.md files, the repository configuration and some security compliance aspects, like the implementation of [Microsoft Identity](https://learn.microsoft.com/entra/identity-platform/v2-overview), there are specifics to an application user interface and business logic, that can only be known to the sample author. + +## e2e tests as a requirement + +When submitting a new template, make sure to have a minimum coverage of the following concerns, using [Playwright](https://playwright.dev/). Our validation pipeline will run the tests and issue a warning when tests fail. + +> [!WARNING] +> Please note that over time, failing tests may result in a template's removal from a collection. + +## Seting up Playwright + +### Devcontainer + +Requires [Playwright VS Code Extension](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright) to be installed via the `.devcontainer` cofiguration. + +``` +"customizations": { + + "vscode": { + + "extensions": [ + + "ms-playwright.playwright" + + ] + + } + + }, +``` + +### Playwright as an application dependency + +Depending on the language, you will need to follow the corresponding instructions. + +- JavaScript - https://playwright.dev/docs/intro + +- Java - https://playwright.dev/java/docs/intro + +- Python - https://playwright.dev/python/docs/intro + +- .NET - https://playwright.dev/dotnet/docs/intro + +**You will also need to follow any other language specific guidelines.** + +## Testing an application deployment to Azure + +We do not require extensive coverage of the application, although the more you’re testing and the higher the coverage, the more reliable your application will be. + +We do require you make sure the application was correctly deployed and is accessible, and that you test any API endpoints to return the correct data payload. + +For example, + +- any and all frontend applications that will be deployed to a hosting service as part of your template +- any and all request to AI Services or databases, that return data. + +### Getting the provisioned URLs + +To effectively test an application after provisioning and deployment, you will need to get the provisioned URLs and run the tests after `azd up` has completed. + +Consider the following GitHub Action configuration, to retrieve a : + +```yaml +name: Run Playwright Tests + +# This workflow will run automatically after azd up is run but before down is run +on: + workflow_run: + workflows: 'pipeline_validation_up_done' + types: + - completed + +jobs: + test: + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v2 + + - name: Set up Node.js + uses: actions/setup-node@v2 + with: + node-version: '20' + + # We need the Azure Developer CLI to be available to get and set the variables + - name: Install Azure Developer CLI + run: | + curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash + npm install -g azure-dev-cli + + - name: Login to Azure + uses: azure/login@v1 + with: + azcliversion: 'latest' + client-id: ${{ secrets.AZURE_CLIENT_ID }} + client-secret: ${{ secrets.AZURE_CLIENT_SECRET }} + tenant-id: ${{ secrets.AZURE_TENANT_ID }} + + # We get the provisioned URLs for the web app hosting and the service endpoint + - name: Get and Save WEBAPP_URL and API_URL to use on the fly + run: | + # Fetching WEBAPP_URL and API_URL from the environment + WEBAPP_URL=$(azd env get-value WEBAPP_URL) + API_URL=$(azd env get-value API_URL) + + - name: Install Dependencies + run: npm install + + - name: Run Playwright Tests + run: npm run test + env: + WEBAPP_URL: ${{ env.WEBAPP_URL }} + API_URL: ${{ env.API_URL }} + +``` + +The example above installs the necessary dependencies, including the Azure Developer CLI, in order to run `azd env get-value` and get the provisioned urls. + +#### Real examples can be found here: + +- [Python e2e Playwright tests action](https://github.com/Azure-Samples/azure-fastapi-postgres-flexible-aca/blob/main/.github/workflows/azure-dev.yml#L73) and [tests](https://github.com/Azure-Samples/azure-fastapi-postgres-flexible-aca/blob/main/src/tests/local/test_playwright.py) +- [JavaScript e2e Playwright tests action](https://github.com/Azure-Samples/azure-search-openai-javascript/blob/main/.github/workflows/playwright.yml) and [tests](https://github.com/Azure-Samples/azure-search-openai-javascript/blob/main/tests/e2e/webapp.spec.ts) + +### Testing the application UI code example with Node.js + +#### Step 1: Make sure the variable is set + +```javascript +// playwright.config.ts or playwright.config.js +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + use: { + baseURL: process.env.WEBAPP_URL || 'http://localhost:3000', // Fallback URL in case WEBAPP_URL is not set + }, +}); +``` + +#### Step 2: Run the test + +The following test checks for the application to be accessible in the provisioned URL and have a title with value "Welcome to the Azure Chatbot template" + +```javascript +// tests/app.spec.ts (for TypeScript) +// or tests/app.spec.js (for JavaScript) + +import { test, expect } from '@playwright/test'; + +test('should display the correct heading on the home page', async ({ page }) => { + // Navigate to the URL provided in the environment variable WEBAPP_URL + await page.goto(process.env.WEBAPP_URL!); + + // Check that the

element contains the correct text + const h1 = await page.locator('h1'); + await expect(h1).toHaveText('Welcome to the Azure Chatbot template'); +}); +``` + +#### Testing a chat endpoint authenticated with MI + +Assume this is a chatGPT like bot, that answers what the weather is like today. + +```javascript +// tests/api.spec.ts (for TypeScript) +// or tests/api.spec.js (for JavaScript) + +import { test, expect, request } from '@playwright/test'; +import axios from 'axios'; + +// Function to get an access token using Managed Identity +async function getAccessToken(resource: string): Promise { + const url = `http://169.254.169.254/metadata/identity/oauth2/token?api-version=2019-08-01&resource=${encodeURIComponent(resource)}`; + const response = await axios.get(url, { + headers: { + 'Metadata': 'true', + }, + }); + + return response.data.access_token; +} + +test('should get a ChatGPT-like response from the API', async () => { + // API URL from environment variable + const apiUrl = process.env.API_URL!; + + // Resource for Azure AD token (this would typically be the API's App ID URI or Resource URI) + const resource = process.env.AZURE_API_RESOURCE || apiUrl; + + // Get the access token using Managed Identity + const token = await getAccessToken(resource); + + // Make the POST request to the API using the access token for authentication + const response = await request.newContext().post(apiUrl, { + headers: { + 'Authorization': `Bearer ${token}`, // Add the Bearer token for authentication + 'Content-Type': 'application/json', + }, + data: { + prompt: "What's the weather like?", // The prompt we are sending to the API + }, + }); + + // Check that the request was successful (status code 200) + expect(response.status()).toBe(200); + + // Parse the JSON response + const responseData = await response.json(); + + // Assuming the API returns a 'response' field with the ChatGPT-like answer + expect(responseData).toHaveProperty('response'); + console.log('ChatGPT-like Response:', responseData.response); + + // Example assertion, modify based on actual API response structure + expect(responseData.response).toContain('weather'); +}); +``` + +These are examples and you will need to update them to meet your usecase and business logic, as well as your constraints. + +We strongly advise you to test all user interfaces and all API endpoints in your application. diff --git a/publishing-guidelines.md b/publishing-guidelines.md index 062922c..95dd3be 100644 --- a/publishing-guidelines.md +++ b/publishing-guidelines.md @@ -109,6 +109,7 @@ Missing standard OSS files | Moderate | Author receives a notification and unpub Missing .devcontainer config | Moderate | Author receives a notification and unpublishing occurs within 7 days Missing pipeling configuration (azure-dev.yml) | Low | Author receives a notification. No unpublishing occurs MI is enabled for all services | Moderate | Author receives a notification and unpublishing occurs within 7 days +Missing required e2e tests | Moderate | Author receives a notification and unpublishing occurs within 15 days ## Give us feedback!