simplify e2e tests to static-only content and single Connect version#3631
simplify e2e tests to static-only content and single Connect version#3631zackverham wants to merge 6 commits intomainfrom
Conversation
Reduce CI matrix from 4 Connect versions to 1 (release), switch all deployments to static HTML only, and remove unused content types (FastAPI, Shiny Python, Quarto, R) and their associated test specs. - Remove 3 spec files: embedded-deployments, error-err-config, open-connect-content - Remove 5 content directories and root-level Python files, keeping only static/ - Switch PCC deployment test from Shiny Python to static HTML - Remove Playwright publish verification (publish-success-task.js) - Remove conditional @pcc tag filtering in CI (always run all tests) - Clean up skipped ShinyApp tests in deployments and positron specs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add tests covering the scenario where a project lives in a subdirectory of the workspace rather than at the root. This closes a coverage gap left by removing the embedded-deployments E2E spec. - bundles: verify bundle from subdirectory only includes that directory's files with correct relative paths, and workspace root files are excluded - bundles: verify file-targeted bundling in a subdirectory resolves baseDir to the file's parent directory - config: verify GetConfigPath, WriteFile/FromFile, and ListConfigFiles all scope correctly to a subdirectory project - initialize: verify GetPossibleConfigs detects content type, sets correct entrypoint/files, and defaults title to subdirectory name Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add tests for the home store's config.active computed properties that detect various configuration error states. These cover the scenarios previously tested only by the deleted error-err-config E2E spec. Tests verify correct state for: - isMissing: config file not found (deleted) - isTOMLError: invalidTOML and unknownTOMLKey errors - isUnknownError: non-TOML agent errors - isUnknownType: config with unknown content type - isEntryMissing: content record with no config name - Valid configuration with no error states active Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use ErrorCode, ContentType, AgentErrorInvalidTOML, AgentErrorTypeUnknown, ConfigurationDetails, and Credential types instead of as any to satisfy the @typescript-eslint/no-explicit-any lint rule. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| # Connect Server versions - tests exclude @pcc tags | ||
| # Simplified to reduce CI time: preview, release, one recent, one old | ||
| connect-version: | ||
| - "preview" # Nightly builds of Connect - catch regressions early | ||
| - "release" # Latest stable release + PCC cloud tests | ||
| - "2025.03.0" # Recent stable version | ||
| - "2023.03.0" # Oldest supported version |
There was a problem hiding this comment.
Do we really want to stop running tests against old connect + preview?
There was a problem hiding this comment.
I'm not sure yet. Still exploring / experimenting on this branch - right now I'm just trying to wrap my head around the shape of the thing.
- Import Credential from credentials.ts, not contentRecords.ts - Add missing productType field to ConfigurationDetails defaults - Use Omit to allow partial configuration overrides in makeConfiguration Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Do we need mock project directories to test a more comprehensive flow of our config -> manifest pipeline? I'm not as familiar with the Go unit tests we have there, but that is another thing these e2e tests do for us, to some degree. |
|
@dotNomad are you thinking of something more than what is starting to take shape in the unit tests? I think the strategy we'd take is just building / tearing down temp directories as part of the test suite. |
I'm seeing some of the bundle texts, for example, check that specific file names exist in the bundle, but right now I'm not seeing tests that verify the I expect that is unit tested in the Go code though. The way that the One of the things I'm worried about with the removal of the e2e tests is dependency scanning and interpreter selection to create the full I'd have to dig in more, so just thinking out loud here. |
Summary
Reduce CI matrix from 4 Connect versions to 1 (release), switch all deployments to static HTML only, and remove unused content types (FastAPI, Shiny Python, Quarto, R) and their associated test specs. Backfill removed E2E coverage with targeted unit tests.
Intent
The E2E test suite tested multiple content types against 4 Connect Server versions, creating a large CI matrix and slow runs. Content-type testing arguably belongs in Connect's own test suite rather than Publisher's. This PR simplifies to static HTML only against one Connect version, while preserving PCC, Workbench, and Positron coverage.
Type of Change
Approach
E2E Simplification
embedded-deployments,error-err-config,open-connect-contentstatic/publish-success-task.js)@pcctag filtering in CI (always run all tests)release)Unit Test Backfill
Added unit tests to cover the key gaps left by removed E2E specs:
Go: Subdirectory path handling (
internal/bundles/bundle_test.go,internal/config/config_test.go,internal/initialize/initialize_test.go)GetConfigPath,WriteFile/FromFile, andListConfigFilesall scope correctly to a subdirectory projectGetPossibleConfigsdetects content type in a subdirectory, sets correct entrypoint/files, and defaults title to subdirectory nameVue: Config error state detection (
extensions/vscode/webviews/homeView/src/stores/home.test.ts)isMissing: config file not found (deleted)isTOMLError:invalidTOMLandunknownTOMLKeyerrors (the exact scenario from the deletederror-err-configE2E)isUnknownError: non-TOML agent errorsisUnknownType: config withtype = "unknown"isEntryMissing: content record with no config nameNet Result
static/)release)What's Preserved
deletePCCContent)Automated Tests
bundles,config, andinitializepackages