Skip to content

simplify e2e tests to static-only content and single Connect version#3631

Draft
zackverham wants to merge 6 commits intomainfrom
worktree-simplify-e2e
Draft

simplify e2e tests to static-only content and single Connect version#3631
zackverham wants to merge 6 commits intomainfrom
worktree-simplify-e2e

Conversation

@zackverham
Copy link
Collaborator

@zackverham zackverham commented Mar 2, 2026

Summary

Reduce CI matrix from 4 Connect versions to 1 (release), switch all deployments to static HTML only, and remove unused content types (FastAPI, Shiny Python, Quarto, R) and their associated test specs. Backfill removed E2E coverage with targeted unit tests.

Intent

The E2E test suite tested multiple content types against 4 Connect Server versions, creating a large CI matrix and slow runs. Content-type testing arguably belongs in Connect's own test suite rather than Publisher's. This PR simplifies to static HTML only against one Connect version, while preserving PCC, Workbench, and Positron coverage.

Type of Change

  • Refactor
  • Tooling

Approach

E2E Simplification

  • Remove 3 spec files: embedded-deployments, error-err-config, open-connect-content
  • Remove 5 content directories and root-level Python files, keeping only static/
  • Switch PCC deployment test from Shiny Python to static HTML
  • Remove Playwright publish verification (publish-success-task.js)
  • Remove conditional @pcc tag filtering in CI (always run all tests)
  • Clean up skipped ShinyApp tests in deployments and positron specs
  • Reduce CI matrix from 4 Connect versions to 1 (release)

Unit Test Backfill

Added unit tests to cover the key gaps left by removed E2E specs:

Go: Subdirectory path handling (internal/bundles/bundle_test.go, internal/config/config_test.go, internal/initialize/initialize_test.go)

  • Bundle from subdirectory only includes that directory's files with correct relative paths; workspace root files are excluded
  • File-targeted bundling in a subdirectory resolves baseDir to the file's parent
  • GetConfigPath, WriteFile/FromFile, and ListConfigFiles all scope correctly to a subdirectory project
  • GetPossibleConfigs detects content type in a subdirectory, sets correct entrypoint/files, and defaults title to subdirectory name

Vue: Config error state detection (extensions/vscode/webviews/homeView/src/stores/home.test.ts)

  • isMissing: config file not found (deleted)
  • isTOMLError: invalidTOML and unknownTOMLKey errors (the exact scenario from the deleted error-err-config E2E)
  • isUnknownError: non-TOML agent errors
  • isUnknownType: config with type = "unknown"
  • isEntryMissing: content record with no config name
  • Happy path: all error states false for a valid configuration

Net Result

Before After
9 E2E spec files 6 E2E spec files
6 content directories + root files 1 directory (static/)
4 Connect versions in CI 1 (release)
Playwright for OAuth + app verification Playwright for OAuth only
PCC deploys Shiny Python PCC deploys static HTML
~36 test runs (9 specs × 4 versions) ~6 test runs (6 specs × 1 version)
No subdirectory path unit tests 7 new Go unit tests
1 Vue store test 9 Vue store tests (+8 for error states)

What's Preserved

  • PCC credential creation (OAuth Device Code via Playwright)
  • PCC deployment workflow (now with static HTML)
  • PCC content cleanup (deletePCCContent)
  • Workbench/Positron deployment (static HTML, unchanged)
  • PCS credential CRUD
  • PCS static deployment
  • OAuth cancellation scenarios
  • Integration Requests PCS vs PCC conditional UI

Automated Tests

  • 7 new Go unit tests across bundles, config, and initialize packages
  • 8 new Vue store tests for config error state detection
  • All existing Go and Vue tests pass

zackverham and others added 5 commits March 2, 2026 11:21
Reduce CI matrix from 4 Connect versions to 1 (release), switch all
deployments to static HTML only, and remove unused content types
(FastAPI, Shiny Python, Quarto, R) and their associated test specs.

- Remove 3 spec files: embedded-deployments, error-err-config, open-connect-content
- Remove 5 content directories and root-level Python files, keeping only static/
- Switch PCC deployment test from Shiny Python to static HTML
- Remove Playwright publish verification (publish-success-task.js)
- Remove conditional @pcc tag filtering in CI (always run all tests)
- Clean up skipped ShinyApp tests in deployments and positron specs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add tests covering the scenario where a project lives in a subdirectory
of the workspace rather than at the root. This closes a coverage gap
left by removing the embedded-deployments E2E spec.

- bundles: verify bundle from subdirectory only includes that directory's
  files with correct relative paths, and workspace root files are excluded
- bundles: verify file-targeted bundling in a subdirectory resolves
  baseDir to the file's parent directory
- config: verify GetConfigPath, WriteFile/FromFile, and ListConfigFiles
  all scope correctly to a subdirectory project
- initialize: verify GetPossibleConfigs detects content type, sets
  correct entrypoint/files, and defaults title to subdirectory name

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add tests for the home store's config.active computed properties that
detect various configuration error states. These cover the scenarios
previously tested only by the deleted error-err-config E2E spec.

Tests verify correct state for:
- isMissing: config file not found (deleted)
- isTOMLError: invalidTOML and unknownTOMLKey errors
- isUnknownError: non-TOML agent errors
- isUnknownType: config with unknown content type
- isEntryMissing: content record with no config name
- Valid configuration with no error states active

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use ErrorCode, ContentType, AgentErrorInvalidTOML, AgentErrorTypeUnknown,
ConfigurationDetails, and Credential types instead of as any to satisfy
the @typescript-eslint/no-explicit-any lint rule.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment on lines -35 to -41
# Connect Server versions - tests exclude @pcc tags
# Simplified to reduce CI time: preview, release, one recent, one old
connect-version:
- "preview" # Nightly builds of Connect - catch regressions early
- "release" # Latest stable release + PCC cloud tests
- "2025.03.0" # Recent stable version
- "2023.03.0" # Oldest supported version
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really want to stop running tests against old connect + preview?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure yet. Still exploring / experimenting on this branch - right now I'm just trying to wrap my head around the shape of the thing.

- Import Credential from credentials.ts, not contentRecords.ts
- Add missing productType field to ConfigurationDetails defaults
- Use Omit to allow partial configuration overrides in makeConfiguration

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dotNomad
Copy link
Collaborator

dotNomad commented Mar 2, 2026

Unit Test Backfill

Do we need mock project directories to test a more comprehensive flow of our config -> manifest pipeline? I'm not as familiar with the Go unit tests we have there, but that is another thing these e2e tests do for us, to some degree.

@zackverham
Copy link
Collaborator Author

@dotNomad are you thinking of something more than what is starting to take shape in the unit tests? I think the strategy we'd take is just building / tearing down temp directories as part of the test suite.

@dotNomad
Copy link
Collaborator

dotNomad commented Mar 2, 2026

are you thinking of something more than what is starting to take shape in the unit tests?

I'm seeing some of the bundle texts, for example, check that specific file names exist in the bundle, but right now I'm not seeing tests that verify the manifest.json we expect to be sent to Connect is correct.

I expect that is unit tested in the Go code though. The way that the bundle_test.go file is setup right now seems like a good path forward to unit testing that behavior when we port it over to TypeScript only.

One of the things I'm worried about with the removal of the e2e tests is dependency scanning and interpreter selection to create the full manifest.json. I believe that is heavily mocked on the Go side right now.

I'd have to dig in more, so just thinking out loud here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants