[Tooling] Add Buildkite release pipelines and fastlane lanes for ReleasesV2#2583
[Tooling] Add Buildkite release pipelines and fastlane lanes for ReleasesV2#2583
Conversation
iangmaia
force-pushed
the
iangmaia/add-studio-to-releasesv2
branch
3 times, most recently
from
06a4903 to
aa320a3
Compare
📊 Performance Test ResultsComparing 575d06e vs trunk site-editor
site-startup
Results are median values from multiple test runs. Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff) |
iangmaia
force-pushed
the
iangmaia/add-studio-to-releasesv2
branch
from
0f541ed to
0b83471
Compare
|
About the critical items:
This was an actual issue due to a change I did last minute last Friday; removing the
This is done in
It does, see this comment. I've implemented other changes related to some of the comments: |
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive CI automation for Studio releases through integration with ReleasesV2, Automattic's release management platform. The automation replaces the previous manual release process with dedicated Fastlane lanes and Buildkite pipelines for each stage of the release lifecycle.
Changes:
- Added 6 Fastlane lanes for release automation:
code_freeze,new_beta_release,finalize_release,publish_release,new_hotfix_release, anddownload_translations - Added Buildkite pipeline configurations for each release stage, triggered by ReleasesV2
- Migrated string extraction from npm script to Fastlane
code_freezelane - Removed deprecated
confirm-tag-matches-version.mjsscript and integrated release builds into separate pipeline - Updated documentation to reflect the new ReleasesV2-based release workflow
Reviewed changes
Copilot reviewed 18 out of 19 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| fastlane/Fastfile | Added comprehensive release management lanes and helper methods; updated distribute_release_build to support default version parameter; improved code style (single quotes, StandardError rescue, removed unused variables) |
| .buildkite/release-pipelines/code-freeze.yml | New pipeline for creating release branches, extracting strings, generating release notes, and creating first beta |
| .buildkite/release-pipelines/new-beta-release.yml | New pipeline for creating subsequent beta releases |
| .buildkite/release-pipelines/finalize-release.yml | New pipeline for creating final release from beta |
| .buildkite/release-pipelines/publish-release.yml | New pipeline for publishing GitHub releases and creating backmerge PRs |
| .buildkite/release-pipelines/download-translations.yml | New pipeline for fetching and committing GlotPress translations |
| .buildkite/release-pipelines/new-hotfix-release.yml | New pipeline for creating hotfix release branches |
| .buildkite/release-build-and-distribute.yml | Extracted release build steps from main pipeline into dedicated release pipeline, triggered by Fastlane lanes |
| .buildkite/pipeline.yml | Removed tag-based release build steps (moved to separate pipeline) |
| .buildkite/commands/checkout-release-branch.sh | New script to checkout release branches from detached HEAD state |
| .buildkite/commands/build-for-windows.ps1 | Removed tag validation check (no longer needed with new workflow) |
| scripts/make-pot.mjs | Removed (functionality moved to Fastlane code_freeze lane) |
| scripts/confirm-tag-matches-version.mjs | Removed (no longer needed with new release workflow) |
| scripts/download-available-site-translations.mjs | Simplified directory creation using recursive: true option |
| package.json | Removed make-pot script (replaced by Fastlane automation) |
| docs/release-process.md | Complete rewrite documenting ReleasesV2 integration and new release lifecycle |
| docs/localization.md | Updated to reflect automated string extraction via code freeze |
| AGENTS.md | Updated release process documentation reference |
| .gitignore | Added temporary github_release_notes.txt file |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Write-Host "Preparing release build..." | ||
| node ./scripts/confirm-tag-matches-version.mjs | ||
| If ($LastExitCode -ne 0) { Exit $LastExitCode } |
There was a problem hiding this comment.
I'm unclear for the rationale about removing those 2 lines, can you elaborate?
There was a problem hiding this comment.
Previously, the release process was based on tags. The first line was a call to a script to check whether the current tag matches the version from the package.json... but since we moved to a release branch based process, this doesn't make a lot of sense. The second line is just checking the exit code from the script.
fastlane/Fastfile
Outdated
| git_add(path: ['./i18n/bundle-strings.pot']) | ||
| git_commit( | ||
| path: ['./i18n/bundle-strings.pot'], |
There was a problem hiding this comment.
Shouldn't we use pot_output variable instead to avoid repeating the path name?
Or is this because using absolute paths (with PROJECT_ROOT_DIR) in this context would not work as expected?
There was a problem hiding this comment.
Yeah, I wasn't sure if the absolute path would work but apparently it is (Fastlane just sends the command to Git); updated on f7708f3.
| current_version = read_package_json_version | ||
| current_beta = current_version[/beta(\d+)$/, 1].to_i | ||
| new_version = "#{version}-beta#{current_beta + 1}" |
There was a problem hiding this comment.
version could contain one version number (say 12.3) while read_package_json_version would return a different base version (e.g. 12.2-beta3), leading to 12.3-beta4 in that case.
This shouldn't happen in theory, at least if the whole release process is run as expected—because new_beta_release should only be triggered after code_freeze has run for the current version, and new_beta_release lane should in theory always run from the release/* branch corresponding of the version value (based on .buildkite/release-pipelines/new-beta-release.yml). But would still be worth adding guardrails for this is case some edge case make us end up in an unexpected situation? (for example if someone accidentally merges trunk back into release/12.3 in an attempt to solve a merge conflict or something, and thus accidentally reverts the version in Package.json to an unexpected value that doesn't match 12.3-*)
There was a problem hiding this comment.
Indeed shouldn't happen but a guard is cheap. Updated on cdbc5a8.
fastlane/Fastfile
Outdated
| # Tag the beta release | ||
| tag_name = "v#{new_version}" | ||
| add_git_tag(tag: tag_name) | ||
| push_git_tags(tag: tag_name) |
There was a problem hiding this comment.
Should we really create the tag before the CI job that builds the beta release has run?
This means that if the beta build fails for some reason (e.g. compilation error or whatnot), we'd have created the git tag even if the beta didn't really get built and might require additional commits to fix the compilation error.
I think it'd be better for the CI job that is triggered by trigger_release_build to do the actual tag (i.e. detect if it's a beta or final build—or being passed that info as a CI env var—and decide to just do a git tag for betas vs a GitHub Release for final builds)
There was a problem hiding this comment.
Good point. It crossed my mind but I preferred to start simple, but it's probably a good idea to handle potential error cases from the start (we could have tag conflicts in case of failures, for example). Updated on b694d47.
I also like the fact that the code to add tag / create release is next to each other; another thing I considered was creating GitHub pre-releases for betas, though the team mentioned we could skip it for now.
fastlane/Fastfile
Outdated
| # Create draft GitHub release with release notes (also creates the tag) | ||
| tag_name = "v#{version}" | ||
| create_github_release( | ||
| repository: GITHUB_REPO, | ||
| version: tag_name, | ||
| target: "release/#{version}", | ||
| release_notes_file_path: release_notes_path, | ||
| release_assets: [], | ||
| prerelease: false, | ||
| is_draft: true | ||
| ) |
There was a problem hiding this comment.
Same thing, it would probably be better to only create the GitHub Release draft at the end of the build being finished. So that the Draft isn't created if the final build fails for some reason (and potentially need to be fixed then re-triggered)
Besides, it would allow us to attach the binaries generated by those builds as assets to the GitHub Release if we wanted to (though probably not, as we publish them to AppsCDN and add the links to AppsCDN as part of append_download_links_to_github_release), and simplify the logic that adds the AppsCDN links in append_download_links_to_github_release if we were to create the GitHub Release draft as a dependent job after all builds have been uploaded to AppsCDN, thus creating the GitHub Release + providing its description with AppsCDN links in one go, instead of creating the GitHub Release in one step, then having to find it using the API during append_download_links_to_github_release to modify its description after the fact
| def extract_release_notes(version:) | ||
| notes_path = File.join(PROJECT_ROOT_FOLDER, 'RELEASE-NOTES.txt') | ||
| content = File.read(notes_path) | ||
|
|
||
| escaped = Regexp.escape(version) | ||
| match = content.match(/^#{escaped}\n=+\n(.*?)(?=^\d+\.\d+\S*\n=+|\z)/m) | ||
|
|
||
| if match | ||
| match[1].strip | ||
| else | ||
| UI.important("No release notes found for #{version} in RELEASE-NOTES.txt") | ||
| "Release #{version}" | ||
| end | ||
| end |
There was a problem hiding this comment.
There was a problem hiding this comment.
The formats are slightly different 😓
- Studio's RELEASE-NOTES.txt format uses
=====underline separators and*bullets:
1.7.4
=====
* Added etc etc ...
* Removed ...
The release toolkit action expects bare version numbers with - bullets and no separator:
1.7.4
- Added etc etc ...
- Removed ...
Related, I think what you mentioned here is spot on: we should probably consolidate these different formats into a single action.
There was a problem hiding this comment.
The release toolkit action expects bare version numbers with - bullets and no separator:
The release-toolkit action expects Actually looking at the code maybe not, you're right. Strange, I could have sworn our other repos using that action used ## in front of the version line iinm, but yeah.## 1.2.3 for section titles. But turns out they expect ---- after the line containing the version to consider it a section title (while here we're using ====). They don't care about the type of bullet point though (- vs *), as any line between the section title and next section is captured verbatim.
I guess we could consider changing the format of the RELEASE-NOTES.txt file while at it as I'm guessing the team is not too strict about it using one or the other? 🤷 And maybe use the occasion to rename the file to the more standard CHANGELOG.md name?
Not a blocker for this PR though
There was a problem hiding this comment.
🎗️ We need to also remember there's some release notes parsing going on in ReleasesV2 😮💨
There was a problem hiding this comment.
Oh TIL I completely forgot about that (I guess it's to generate the Release Summary P2 post, right?).
Good catch and good point 👍
Also notice how the format expected by the parsing logic in ReleasesV2 is the same format than what extract_release_notes_for_version, namely expecting ---- instead of ==== for section separators.
fastlane/Fastfile
Outdated
| # Uses GitHub's generate-notes API to list PRs since the last published release. | ||
| def update_release_notes_draft(version:) | ||
| # Find the latest stable release tag reachable from HEAD to use as the starting point | ||
| previous_tag = find_previous_tag(pattern: 'v*') |
There was a problem hiding this comment.
❌ This will also include the previous betas in the search. I don't think that's what we want?
Looking at the existing RELEASE-NOTES.txt file it seems that new sections are only added per production release, not for each beta. And this update_release_notes_draft helper is also only called during code_freeze, so that confirms that we want the list of PRs since last release, not since the last intermediate beta.
If we want to keep this same behavior, we thus need to compare the PRs with the last final release tag, excluding beta tags.
I don't think our find_previous_tag action currently support custom --exclude patterns in order to avoid batching beta tags like v1.7.4-beta3. So either we need to make a PR in release-toolkit add this feature to find_previous_tag (which would probably be useful to make it more flexible in general anyway), or we need to start adopting a different tag naming convention (e.g. like we do with Tumblr and prefixing them with "folder names" e.g.beta/1.7.4-beta3 vs final/1.7.4), so that we can write a find_previous_tag(pattern:…) that only matches final tags.
There was a problem hiding this comment.
Nice catch!
Updated on dcb9cf8 + 93bfe7f, I think this should work? -- I wanted to avoid changing the tag convention (though it's probably not a big deal). I've created AINFRA-2056 to add this improvement to release-toolkit.
There was a problem hiding this comment.
Yep I think it should work 👍
But thinking about it I think right now only Tumblr uses this action which is why with its tag naming convention using folder names it doesn't have this limitation… but all our other products would, just like Studio here? Eg most of our other apps use a x.y-rc-N version for their betas and would have the same problem (but it just happens that they don't use generated change logs based on PRs list and get_previous_tag yet hence why we never realized about that limitation until now.
So AINFRA-2056 would definitely be useful not just for Studio in fact!
| ) | ||
|
|
||
| # Convert "* Title by @user in https://github.com/Org/Repo/pull/123" to "* Title #123" | ||
| lines = changelog.lines.reject { |l| l.start_with?('##') || l.strip.empty? } |
There was a problem hiding this comment.
.github/releases.yml config file in the future, so that get_prs_between_tags groups PRs in different sections based on GitHub Labels (for other products use this to differentiate e.g. "Bug Fixes" vs "New Feature" PRs and separate them in the release notes), this code would have the side effect of removing those subsections.
This is not a problem right now as there's no .github/releases.yml file configured yet and their RELEASE-NOTES.txt file is listing a flat list of PRs per release. But to keep in mind if they want to consider their release notes to list changes classified by different kinds in the future.
| new_section = "#{version}\n=====\n#{formatted}\n" | ||
|
|
||
| # Insert the new version section after the "Unreleased" header | ||
| unless content.sub!(/^(Unreleased\n=+\n)\n*/, "\\1\n#{new_section}\n") | ||
| UI.user_error!("Could not find 'Unreleased' header in RELEASE-NOTES.txt. Please ensure the file has an 'Unreleased' section.") | ||
| end | ||
| File.write(notes_path, content) |
There was a problem hiding this comment.
I thought we had a dedicated common action to handle those insertions in the right place in RELEASE-NOTES.txt markdown-formatted files, but maybe not (I didn't find it in common/* actions of release-toolkit at least). Would be nice to work on better actions in release-toolkit to manipulate Markdown-formatted files like those release-notes.txt / CHANGELOG.md files that most of our repos have to consolidate the implementation and logic across all of them without having to re-invent the wheel every time.
There was a problem hiding this comment.
Indeed. I've created AINFRA-2077 to handle it at some point.
| ### 4. Finalize Release | ||
|
|
||
| **ReleasesV2 milestone**: Release | **Fastlane lane**: `finalize_release` | ||
|
|
||
| - Removes beta suffix (sets version to `<version>`) | ||
| - Creates a **draft** GitHub release with notes from `RELEASE-NOTES.txt` | ||
| - Triggers the final release build, which uploads to the Apps CDN, appends download links to the draft GitHub release, and notifies Slack | ||
|
|
||
| ### 5. Publish Release | ||
|
|
||
| **Fastlane lane**: `publish_release` | ||
|
|
||
| - Publishes the draft GitHub release | ||
| - Creates a backmerge PR from `release/<version>` into `trunk` |
There was a problem hiding this comment.
Just to validate with the team: is this split between "finalize" and "publish" intentional and useful to you?
In all our other release processes for mobile apps this makes sense because the apps need to be submitted for review and approved by Apple / Google before we can publish them. So we finalize the release to submit it to Apple/Google for review in one step, let them review and hopefully approve the build, and once we're ready we publish to end users.
But with Studio being distributed outside of the AppStores and not dependent on Apple/Google review/validation of each build, I wonder if this split is necessary.
Though to be honest I still think it'd be useful, if only for the release manager and your team to at least smoke-test the release build before publishing the GitHub Release officially.
But given the "Finalize Release" stage already uploads the final build to AppsCDN—which would make it publicly available and add it to the Sparkle RSS feed thus being proposed as a new version update to users via Sparkle even before you go through the Publish Release stage, that raises the question if this is useful, and if it is, if we wouldn't need an additional mechanism that makes builds uploaded to AppsCDN via Finalize Release would upload it as private at first, and "Publish Release" would change the status from private to public once you had the opportunity to smoke-test the final build.
There was a problem hiding this comment.
I like that idea! Uploading it as private in the Finalize Release step and then making it public in the Publish Release step is really smart. It’s also consistent with the name of each stage.
There was a problem hiding this comment.
[❓] @hannahtinkler: Is there already an API to change the visibility of a binary already uploaded to AppsCDN? I looked at the FG Technical Documentation and didn't see such an API mentioned but maybe it exists and is just not yet documented, or maybe it's just already implemented as a regular/generic WP API call if it's just changing attributes of a post anyway like it's possible for regular posts?
By the way I'm not sure if the best approach here would be to play with:
- The
visibilityattribute—starting withInternal, then change it toExternalonce ready—to make the binary published and easily testable by a12s internally before making it public? - Or the
post_statusattribute—starting withdraftstatus while a12s are smoke-testing the build, then change it topublish?
I'm not 100% sure of the consequence of one vs the other.
- I'm guessing that if we're using
post_statuswe can probably already use the official WP API to change the status later like we'd do for any post or page, so at least the API is already there… but how will this reflect on a binary uploaded to AppsCDN, will it be downloadable/testable internally easily still, or will it risk not reflect well enough on how the download experience will look on the final product once published? (Depending on if the RM who wants to smoke-test it wants to test it via the Internal track of Sparkle vs manual install…) - While
visibilityis probably a better one for testability (because a build that is published but withvisibility: Internalis, by design, easier to download and test for a12s, since that's what theInternaltrack is for after all)… but since that's a custom attribute of the taxonomy, does this mean it will require a specific API to be changed (and for AppsCDN to account for that change, regenerate the Sparkle RSS XML if needed, and whatnot)?
There was a problem hiding this comment.
There's no dedicated endpoint to change a build's visibility, but it's just post meta, so you should be able to use the REST API? 🤔
does this mean it will require a specific API to be changed (and for AppsCDN to account for that change, regenerate the Sparkle RSS XML if needed, and whatnot)?
This is all dynamically generated from the post data, so should be fine 🙂
There was a problem hiding this comment.
Thanks Hannah! 👍
I've created AINFRA-2102 to handle the visibility update as a follow-up.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>
There was a problem hiding this comment.
@iangmaia, thanks for creating this PR and moving forward with the Releases V2 migration. The code looks good, and I’m looking forward to testing it together.
I’ve only spotted a misleading step: download-available-site-translations.mjs is not downloading the translations for WordPress Studio. Currently, this is a manual step, and here is an example PR #2557
| "test:watch": "vitest", | ||
| "e2e": "npx playwright install && npx playwright test", | ||
| "test:metrics": "npx playwright test --config=./tools/metrics/playwright.metrics.config.ts", | ||
| "make-pot": "node ./scripts/make-pot.mjs", |
There was a problem hiding this comment.
Could we keep the make-pot script and file in case we want to upload them manually? Or will be possible to trigger it from Buildkite?
There was a problem hiding this comment.
What make-pot was doing is in this PR in code_freeze, but I created a follow-up PR to extract it so it can be executed separately: #2658, so you should be able to run bundle exec fastlane generate_pot_file -- the POT file now living in the repo.
fastlane/Fastfile
Outdated
| # Download the latest translations and create a PR to merge them into the release branch. | ||
| # | ||
| # @param skip_confirm [Boolean] Skip interactive confirmation prompts (default: false) | ||
| # | ||
| lane :download_translations do |skip_confirm: false, github_username: nil| | ||
| current_branch = Fastlane::Helper::GitHelper.current_git_branch | ||
|
|
||
| UI.important <<~PROMPT | ||
| Downloading translations. This will: | ||
| - Download latest translations from GlotPress | ||
| - Create a PR to merge them into `#{current_branch}` | ||
| PROMPT | ||
| next unless skip_confirm || UI.confirm('Continue?') | ||
|
|
||
| sh('node', File.join(PROJECT_ROOT_FOLDER, 'scripts', 'download-available-site-translations.mjs')) |
There was a problem hiding this comment.
I executed the script node scripts/download-available-site-translations.mjs and it downloaded the WordPress core translations in the path wp-files/latest/available-site-translations.json which is ignored by git.
I think we need to create a script to download the WordPress translations from WPcom Studio project.
There was a problem hiding this comment.
Ahh, I think this is probably a legacy script then and I probably got confused? I see that https://github.com/Automattic/studio/blob/trunk/docs/localization.md mentions a manual export. I'll fix it quickly!
There was a problem hiding this comment.
Done, thanks for catching it! Updated on 7da895556e0e5a7646ce67ce86d7b342595a035b. Just tested it here and it worked well -- we can double check once we run a real test later today.
iangmaia
force-pushed
the
iangmaia/add-studio-to-releasesv2
branch
from
a3cf935 to
575d06e
Compare
Fixes AINFRA-1806, AINFRA-1943
ReleasesV2 PR: https://github.a8c.com/Automattic/wpcom/pull/203624
Proposed changes
Adds CI automation for Studio releases, to be triggered from ReleasesV2. This is the companion PR to the wpcom ReleasesV2 configuration PR.
Fastlane lanes added
code_freeze— Creates release branch from trunk, extracts translatable strings, delegates tonew_beta_releasenew_beta_release— Bumps beta version, commits, pushes, creates GitHub prerelease, triggers release buildfinalize_release— Bumps to final version, creates draft GitHub release, triggers release buildpublish_release— Publishes draft GitHub release, creates backmerge PR to trunknew_hotfix_release— Creates hotfix branch from latest release tag, bumps versiondistribute_release_build— Distributes build artifacts (called from release-build-and-distribute pipeline)Buildkite pipelines added
release-pipelines/code-freeze.ymlrelease-pipelines/new-beta-release.ymlrelease-pipelines/finalize-release.ymlrelease-pipelines/publish-release.ymlrelease-pipelines/download-translations.ymlrelease-pipelines/new-hotfix-release.ymlrelease-build-and-distribute.yml— Build jobs extracted from the mainpipeline.ymland triggered by Fastlane lanes to build Mac/Windows artifacts and distribute.How it works
ReleasesV2 triggers a Buildkite build with a
PIPELINEenv var pointing to the release pipeline YAML added in this PR. The release pipeline runs the corresponding Fastlane lane.For lanes that produce builds (
new_beta_release,finalize_release), we usebuildkite_add_trigger_stepto trigger a separaterelease-build-and-distributebuild that handles Mac/Windows builds, artifact distribution and release tagging / GitHub release creation.Testing instructions
code_freeze) locally (withoutskip_confirm) to verify prompts and flow