Skip to content

feat(vertexai): add integration test recordings for vertexai provider#5219

Draft
Artemon-line wants to merge 3 commits intollamastack:mainfrom
Artemon-line:feat/vertexai-integration-recordings
Draft

feat(vertexai): add integration test recordings for vertexai provider#5219
Artemon-line wants to merge 3 commits intollamastack:mainfrom
Artemon-line:feat/vertexai-integration-recordings

Conversation

@Artemon-line
Copy link
Copy Markdown
Contributor

@Artemon-line Artemon-line commented Mar 19, 2026 •
edited
Loading

Summary

Adds vertexai to the CI responses test suite, enabling integration testing of the vertexai provider. Recordings will be auto-generated by the CI recording workflow after the WIF authentication PR merges.

  • Add vertexai setup definition to tests/integration/suites.py with publishers/google/models/gemini-2.0-flash
  • Add vertexai entry to CI matrix (ci_matrix.json)
  • Register vertexai model and provider in ci-tests distribution template (ci_tests.py) so config.yaml is generated correctly by distro-codegen
  • Add google-genai SDK patching to api_recorder.py for record/replay (patches AsyncModels.generate_content, generate_content_stream, embed_content)
  • Add vertexai-specific test skips for unsupported features (logprobs, service_tier, file search filters, incomplete_details length)
  • Add CI workflow env vars for vertexai in integration-tests.yml
  • Add VERTEX_AI_PROJECT / VERTEX_AI_LOCATION env var passthrough in integration-tests.sh

Closes #5102

Blocked by: #5276 — GCP Workload Identity Federation needed for Vertex AI CI authentication. After #5276 merges, the recording workflow will auto-generate and commit vertexai recordings via commit-recordings.yml.

Test Plan

  • Full responses test suite passes against live Vertex AI: 107 passed, 117 skipped, 0 errors
  • Pre-commit hooks pass (distro-codegen, ruff, mypy)
  • Unit tests pass (1777 passed)
  • CI replay mode runs successfully after recordings are auto-generated by the recording workflow

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Mar 19, 2026
@cdoern
Copy link
Copy Markdown
Collaborator

cdoern commented Mar 19, 2026

@Artemon-line if you're going to be adding integration tests, please also add vertex to the auto re-record mechanism that was just introduced

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 19, 2026 •
edited
Loading

✅ Recordings committed successfully

Recordings from the integration tests have been committed to this PR.

View commit workflow

@Artemon-line Artemon-line marked this pull request as draft March 19, 2026 18:53
@Artemon-line Artemon-line force-pushed the feat/vertexai-integration-recordings branch from 60e4f77 to 8dcb7af Compare March 19, 2026 19:13
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 23, 2026

This pull request has merge conflicts that must be resolved before it can be merged. @Artemon-line please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Mar 23, 2026
@Artemon-line Artemon-line force-pushed the feat/vertexai-integration-recordings branch 2 times, most recently from 266ebef to 4fa9772 Compare March 23, 2026 15:52
@mergify mergify bot removed the needs-rebase label Mar 23, 2026
@Artemon-line Artemon-line force-pushed the feat/vertexai-integration-recordings branch 2 times, most recently from cab9442 to 576c237 Compare March 23, 2026 17:25
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 23, 2026

This pull request has merge conflicts that must be resolved before it can be merged. @Artemon-line please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@Artemon-line Artemon-line marked this pull request as ready for review March 24, 2026 14:39
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 24, 2026

This pull request has merge conflicts that must be resolved before it can be merged. @Artemon-line please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Mar 24, 2026
@Artemon-line
Copy link
Copy Markdown
Contributor Author

Artemon-line commented Mar 24, 2026

@Artemon-line if you're going to be adding integration tests, please also add vertex to the auto re-record mechanism that was just introduced

Seems like we need to implement this first: #5272

@Artemon-line Artemon-line marked this pull request as draft March 24, 2026 18:27
@Artemon-line
feat(vertexai): add CI test infrastructure for vertexai provider
4bf3219 
Add vertexai to the CI responses test suite, enabling integration testing
of the vertexai provider. Recordings will be auto-generated by the CI
recording workflow after the WIF authentication PR (llamastack#5276) merges.

- Add vertexai setup definition to tests/integration/suites.py
- Add vertexai entry to CI matrix (ci_matrix.json)
- Register vertexai model and provider in ci-tests distribution template
- Add google-genai SDK patching to api_recorder.py for record/replay
- Add vertexai-specific test skips for unsupported features (logprobs,
  service_tier, file search filters, incomplete_details length)
- Add CI workflow env vars for vertexai in integration-tests.yml
- Add VERTEX_AI env var passthrough in integration-tests.sh

Closes llamastack#5102

Signed-off-by: Artemy Hladenko <ahladenk@redhat.com>
Signed-off-by: Artemy <ahladenk@redhat.com>
@Artemon-line Artemon-line force-pushed the feat/vertexai-integration-recordings branch from b6ad0bc to 4bf3219 Compare March 27, 2026 14:49
@Artemon-line
chore: merge upstream/main and resolve suites.py conflict
fdeb5ca 
Resolve conflict in tests/integration/suites.py caused by tgi setup
removal upstream. Keep vertexai setup definition, drop removed tgi entry.

Signed-off-by: Artemy Hladenko <ahladenk@redhat.com>
Signed-off-by: Artemy <ahladenk@redhat.com>
@mergify mergify bot removed the needs-rebase label Mar 27, 2026
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 27, 2026

This pull request has merge conflicts that must be resolved before it can be merged. @Artemon-line please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Mar 27, 2026
@Artemon-line
chore: merge upstream/main and resolve test_file_search.py conflict
6ca3bde 
Keep vertexai skip and adopt upstream watsonx skip message update.

Signed-off-by: Artemy Hladenko <ahladenk@redhat.com>
Signed-off-by: Artemy <ahladenk@redhat.com>
@mergify mergify bot removed the needs-rebase label Mar 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashwinb ashwinb Awaiting requested review from ashwinb ashwinb is a code owner

@raghotham raghotham Awaiting requested review from raghotham raghotham is a code owner

@ehhuang ehhuang Awaiting requested review from ehhuang ehhuang is a code owner

@leseb leseb Awaiting requested review from leseb leseb is a code owner

@bbrowning bbrowning Awaiting requested review from bbrowning bbrowning is a code owner

@mattf mattf Awaiting requested review from mattf mattf is a code owner

@franciscojavierarceo franciscojavierarceo Awaiting requested review from franciscojavierarceo franciscojavierarceo is a code owner

@cdoern cdoern Awaiting requested review from cdoern cdoern is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(vertexai): add integration test recordings for vertexai provider

2 participants

@Artemon-line @cdoern