-
Notifications
You must be signed in to change notification settings - Fork 565
Bump to NDK r29 #9926
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
grendello
wants to merge
28
commits into
main
Choose a base branch
from
dev/grendel/ndk-r29
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Bump to NDK r29 #9926
Changes from all commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
04472c2
Bump to NDK r29-beta1
grendello 4d2d1de
Bump to r29-beta2
grendello 19c1302
Bump to beta3
grendello c148cf8
Fix after rebase
grendello 0feca1e
Bump to the release version
grendello 979cc46
Disable linking against libc++
grendello b2829a8
libc++ bits and pieces
grendello e4071fd
Still broken, baby steps
grendello 18847bf
Not there
grendello 060b2ac
Better, but still broken. TBC tomorrow
grendello 764866c
More libc++ imported. Build still broken.
grendello eeda47a
Compiles and links without errors. Not tested at runtime yet.
grendello 6454ea0
Not needed
grendello dbadad3
Fix 2x oops
grendello 88afb2a
A portion of `std::format` instances removed for NativeAOT
grendello c333138
More std::format eliminated from the NativeAOT host
grendello 2008803
NativeAOT sample links correctly now
grendello 394608b
Beginnings of an LLVM sources update utility
grendello 133b26e
Stage
grendello 0598406
Docs and some finishing touches to the updater
grendello 7a37358
Update apkdesc files
grendello 6a795bd
Should fix the multi-project NativeAOT builds
grendello bd7e9b2
Always specify configuration when building a solution
grendello 22f5512
Ugh
grendello ef0ad43
Ugh
grendello ed80f69
Fix after rebase
grendello eebfa77
Unbreak the BuildSolutionWithMultipleProjectsInParallel test for Core…
grendello 468f41b
Update apkdesc files
grendello File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # How to update Android NDK | ||
|
|
||
| For the most part, update of the NDK version used to build this repository is | ||
| very straightforward. The only complication arises from the fact that we carry | ||
| a copy of some LLVM source files, for its libc++ and libc++abi libraries. | ||
| The copied files are needed only by the `NativeAOT` host (see https://github.com/dotnet/runtime/issues/121172), | ||
| the `MonoVM` and `CoreCLR` hosts link against the two libraries directly. | ||
|
|
||
| Our copy of LLVM sources *must* be updated *every time* we update the NDK version. | ||
|
|
||
| ## Update NDK reference in `xaprepare` | ||
|
|
||
| Visit https://developer.android.com/ndk/downloads/index.html to obtain NDK revision | ||
| information then edit the `build-tools/xaprepare/xaprepare/ConfigAndData/BuildAndroidPlatforms.cs` | ||
| file and update the `BuildAndroidPlatforms.AndroidNdkVersion` and `BuildAndroidPlatforms.AndroidNdkPkgRevision` | ||
| properties with the information obtained from the NDK distribution URL. | ||
|
|
||
| ## Update LLVM sources | ||
|
|
||
| The best way to do it is by using the `tools/update-llvm-sources` utility, after runing `xaprepare`. | ||
|
|
||
| You can run the utility directly with `dotnet tools/update-llvm-sources` or, if you are on a Unix | ||
| system, run `make update-llvm` from the top directory. | ||
|
|
||
| ### Details (should you need to update sources manually) | ||
|
|
||
| Android NDK uses a fork of the upstream LLVM repository, currently | ||
| https://android.googlesource.com/toolchain/llvm-project and this is the repository updated tool | ||
| mentioned above uses to fetch the files. | ||
|
|
||
| Android NDK has a manifest file for the LLVM toolchain which enumerates revisions of all the | ||
| components, however that file changes name in each release, based on information it yet another | ||
| manifest file, namely `${ANDROID_NDK_ROOT}/BUILD_INFO`. This is a JSON file, which contains a | ||
| number of properties, we are however interested only in one of them, named `bid`. Its value | ||
| is a string which is part of the second manifest, found in the `${ANDROID_NDK_ROOT}/manifest_${bid}.xml` | ||
| file. | ||
|
|
||
| In the XML manifest, we can find an element named `project`, with its `name` attribute set to | ||
| `toolchain/llvm-project` - the `revision` attribute of that element is the Git revision we need | ||
| in order to access sources from the Google's `llvm-project` fork. | ||
|
|
||
| Once you have the revision, you can either clone the Android fork repository and checkout the | ||
| revision, or visit the individual files in the browser. All the LLVM sources we copied are | ||
| contained in the `src-ThirdParty/llvm/` directory, with the subdirectories reflecting exactly | ||
| the `llvm-project` layout. This way, you can take a file path relative to `src-ThirdParty/llvm` and | ||
| form the file's URL as follows: | ||
|
|
||
| ``` | ||
| https://android.googlesource.com/toolchain/llvm-project/+/${LLVM_REVISION}/${RELATIVE_FILE_PATH} | ||
| ``` | ||
|
|
||
| Visiting this url will show you the file with syntax higlighting and line numbers, however it's | ||
| not the raw source, but rather its HTML rendering, useless for our purpose. In order to fetch the | ||
| raw source, we need to append `?format=TEXT` to the URL. Once visited in the browser (or fetched | ||
| using `curl` or `wget`), the resulting file will be downloaded but not yet ready for updating of | ||
| our copy. The downloaded file is encoded in the `base64` encoding and must be decoded before use. | ||
|
|
||
| On Unix systems this can be done using the following command: | ||
|
|
||
| ```shell | ||
| $ base64 -d < downloaded_file.cpp > file.cpp | ||
| ``` | ||
|
|
||
| After that, the resulting file can be copied to its destination in our source tree. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| using System; | ||
|
|
||
| namespace Xamarin.Android.Tools; | ||
|
|
||
| static class LlvmUpdateInfo | ||
| { | ||
| public const string Revision = "@LLVM_PROJECT_REVISION@"; | ||
| public const string Version = "@LLVM_PROJECT_VERSION@"; | ||
| public static readonly Uri BaseUrl = new Uri ("@LLVM_PROJECT_BASE_URL@"); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
185 changes: 185 additions & 0 deletions
185
build-tools/xaprepare/xaprepare/Steps/Step_Generate_LLVM_UpdateInfo.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,185 @@ | ||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.IO; | ||
| using System.Text; | ||
| using System.Text.Json; | ||
| using System.Threading.Tasks; | ||
| using System.Xml; | ||
|
|
||
| namespace Xamarin.Android.Prepare; | ||
|
|
||
| class Step_Generate_LLVM_UpdateInfo : Step | ||
| { | ||
| static ReadOnlySpan<byte> Utf8Bom => new byte[] { 0xEF, 0xBB, 0xBF }; | ||
| static readonly byte[] BidPropertyName = Encoding.UTF8.GetBytes ("bid"); | ||
|
|
||
| public Step_Generate_LLVM_UpdateInfo () | ||
| : base ("Generating LLVM source update information") | ||
| {} | ||
|
|
||
| #pragma warning disable CS1998 | ||
| protected override async Task<bool> Execute (Context context) | ||
| { | ||
| try { | ||
| if (!Generate (context)) { | ||
| Log.WarningLine ("Failed to generate LLVM update info. Attempt to update LLVM sources may fail."); | ||
| } | ||
| } catch (Exception ex) { | ||
| Log.WarningLine ($"Failed to generate LLVM update info. {ex.Message}"); | ||
| Log.DebugLine ($"Exception was thrown while generating LLVM update info."); | ||
| Log.DebugLine (ex.ToString ()); | ||
| } | ||
|
|
||
| // This step isn't critical, we never fail. | ||
| return true; | ||
| } | ||
| #pragma warning restore CS1998 | ||
|
|
||
| bool Generate (Context context) | ||
| { | ||
| // BUILD_INFO is a JSON document with build information, we need the "bid" component from there as it forms | ||
| // part of the toolchain manifest name | ||
| string? bid = GetBid (Path.Combine (Configurables.Paths.AndroidToolchainRootDirectory, "BUILD_INFO")); | ||
| if (String.IsNullOrEmpty (bid)) { | ||
| Log.DebugLine ("Unable to find LLVM toolchain bid information."); | ||
| return false; | ||
| } | ||
|
|
||
| // Manifest contains GIT revisions of various NDK components. We need the LLVM project's one from there. | ||
| string toolchainManifestPath = Path.Combine (Configurables.Paths.AndroidToolchainRootDirectory, $"manifest_{bid}.xml"); | ||
| (string? llvmProjectPath, string? llvmProjectRevision) = GetLlvmProjectInfo (toolchainManifestPath); | ||
|
|
||
| if (String.IsNullOrEmpty (llvmProjectPath)) { | ||
| Log.DebugLine ("Failed to read LLVM project path from the manifest."); | ||
| return false; | ||
| } | ||
|
|
||
| if (String.IsNullOrEmpty (llvmProjectRevision)) { | ||
| Log.DebugLine ("Failed to read LLVM project GIT revision from the manifest."); | ||
| return false; | ||
| } | ||
|
|
||
| string? llvmProjectVersion = null; | ||
| string androidVersionPath = Path.Combine (Configurables.Paths.AndroidToolchainRootDirectory, "AndroidVersion.txt"); | ||
| if (Path.Exists (androidVersionPath)) { | ||
| try { | ||
| foreach (string line in File.ReadLines (androidVersionPath)) { | ||
| // In NDK r29 LLVM version was on the first line | ||
| llvmProjectVersion = line.Trim (); | ||
| break; | ||
| } | ||
| } catch (Exception ex) { | ||
| Log.DebugLine ($"Failed to read LLVM Android version file '{androidVersionPath}'"); | ||
| Log.DebugLine ("Exception was thrown:"); | ||
| Log.DebugLine (ex.ToString ()); | ||
| } | ||
| } else { | ||
| Log.WarningLine ($"LLVM Android version file not found at {androidVersionPath}"); | ||
| } | ||
|
|
||
| if (String.IsNullOrEmpty (llvmProjectVersion)) { | ||
| llvmProjectVersion = "<unknown>"; | ||
| } | ||
|
|
||
| Log.InfoLine ("LLVM project path: ", llvmProjectPath); | ||
| Log.InfoLine ("LLVM project revision: ", llvmProjectRevision); | ||
| Log.InfoLine ("LLVM project version: ", llvmProjectVersion); | ||
|
|
||
| // Manifest uses https://googleplex-android.googlesource.com/ which is not accessible for mere mortals, | ||
| // therefore we need to use the public URL | ||
| var baseURIBuilder = new UriBuilder (Configurables.Urls.GoogleSourcesBase); | ||
| baseURIBuilder.Path = $"{llvmProjectPath}/+/{llvmProjectRevision}"; | ||
| Uri baseURI = baseURIBuilder.Uri; | ||
|
|
||
| const string updateSourcesInputName = "LlvmUpdateInfo.cs.in"; | ||
| string updateInfoSourceInputPath = Path.Combine (Configurables.Paths.BuildToolsScriptsDir, updateSourcesInputName); | ||
| string updateInfoSourceOutputPath = Path.Combine (Configurables.Paths.BuildBinDir, Path.GetFileNameWithoutExtension (updateSourcesInputName)); | ||
|
|
||
| Log.InfoLine (); | ||
| Log.InfoLine ($"Generating LLVM update info sources."); | ||
| var updateInfoSource = new GeneratedPlaceholdersFile ( | ||
| new Dictionary <string, string> (StringComparer.Ordinal) { | ||
| { "@LLVM_PROJECT_BASE_URL@", baseURI.ToString () }, | ||
| { "@LLVM_PROJECT_REVISION@", llvmProjectRevision }, | ||
| { "@LLVM_PROJECT_VERSION@", llvmProjectVersion }, | ||
| }, | ||
| updateInfoSourceInputPath, | ||
| updateInfoSourceOutputPath | ||
| ); | ||
| updateInfoSource.Generate (context); | ||
|
|
||
| return true; | ||
| } | ||
|
|
||
| (string? path, string? revision) GetLlvmProjectInfo (string manifestPath) | ||
| { | ||
| Log.DebugLine ($"Reading LLVM toolchain manifest from '{manifestPath}'"); | ||
|
|
||
| if (!File.Exists (manifestPath)) { | ||
| Log.DebugLine ($"NDK LLVM manifest '{manifestPath}' not found"); | ||
| return (null, null); | ||
| } | ||
|
|
||
| var readerSettings = new XmlReaderSettings { | ||
| ValidationType = ValidationType.None, | ||
| DtdProcessing = DtdProcessing.Ignore, | ||
| IgnoreWhitespace = true, | ||
| IgnoreComments = true, | ||
| IgnoreProcessingInstructions = true, | ||
| }; | ||
| using var reader = XmlReader.Create (manifestPath, readerSettings); | ||
| var doc = new XmlDocument (); | ||
| doc.Load (reader); | ||
|
|
||
| XmlNode? llvmToolchain = doc.SelectSingleNode ("//manifest/project[@name='toolchain/llvm-project']"); | ||
| if (llvmToolchain == null) { | ||
| Log.DebugLine ("Failed to find LLVM toolchain info in the manifest."); | ||
| return (null, null); | ||
| } | ||
|
|
||
| if (llvmToolchain.Attributes == null) { | ||
| Log.DebugLine ("Unable to read path and revision info about the LLVM toolchain, no attributes on the element."); | ||
| return (null, null); | ||
| } | ||
|
|
||
| XmlAttribute? path = llvmToolchain.Attributes["path"]; | ||
| XmlAttribute? revision = llvmToolchain.Attributes["revision"]; | ||
|
|
||
| return (path?.Value, revision?.Value); | ||
| } | ||
|
|
||
| string? GetBid (string buildInfoPath) | ||
| { | ||
| Log.DebugLine ($"Reading LLVM toolchain build info from '{buildInfoPath}'"); | ||
|
|
||
| ReadOnlySpan<byte> manifestBytes = File.ReadAllBytes (buildInfoPath); | ||
|
|
||
| if (manifestBytes.StartsWith (Utf8Bom)) { | ||
| manifestBytes = manifestBytes.Slice (Utf8Bom.Length); | ||
| } | ||
|
|
||
| string? bid = null; | ||
| var reader = new Utf8JsonReader (manifestBytes); | ||
| while (reader.Read ()) { | ||
| if (reader.TokenType != JsonTokenType.PropertyName) { | ||
| continue; | ||
| } | ||
|
|
||
| if (!reader.ValueTextEquals (BidPropertyName)) { | ||
| continue; | ||
| } | ||
|
|
||
| // let's assume the manifest document is formatted correctly | ||
| reader.Read (); | ||
| if (reader.TokenType != JsonTokenType.String) { | ||
| Log.DebugLine ($"Invalid token type '{reader.TokenType}' for the 'bid' property in LLVM manifest."); | ||
| return null; | ||
| } | ||
|
|
||
| bid = reader.GetString (); | ||
| break; | ||
| } | ||
|
|
||
| return bid; | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@grendello just curious, how often should we do this?
Every release? or just every major one?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tend to track the latest stable release from https://developer.android.com/ndk/downloads/index.html, it is updated every few months.