Improved getDependency workflow #758

Amrutha-Kanhirathingal · 2025-10-24T14:42:45Z

getDependencies.pl script run twice.Once with customUrl. If it failed, retry with 3rd party URL.
Closes: Improve getDependency workflow aqa-tests#6461

Amrutha-Kanhirathingal · 2025-10-24T15:52:11Z

Logs :

System jars are already present in machine

https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55715/console

Added wrong url to fail downloading jar using custom url (FYI -the log and snapshot has old log statements which may not be matching with the current pr logs)

https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55707/consoleFull

Extended.Perf : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55703/ : completed
sanity.functional : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55739/consoleFull: failed
extended.functional : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55723/: UNSTABLE
special.functional : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55724 : completed
sanity.perf : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55722 : completed

llxia · 2025-10-24T20:05:32Z

I do not see that a customUrl is set and used for non-system tests. Please check descriptions in adoptium/aqa-tests#6461 (comment)

00:03:06.008  Custom URL: https://openj9-jenkins.osuosl.org/job/test.getDependency/lastSuccessfulBuild/artifact/

00:03:34.431       [exec] Downloading jar from third-party URL for json-simple.jar: https://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar

sanity.functional : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55718/

Only ran one test. It did not run entire sanity.functional.

Amrutha-Kanhirathingal · 2025-10-28T18:46:50Z

Hi @llxia ,

As per discussion i have updated the script to consider non-system tests as well,
Please find the logs

special function : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55803/
extended perf : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55802/
sanity.function : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55801/parameters/
special function -failed case : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55812/console

Thanks

llxia · 2025-10-29T20:58:11Z

scripts/getDependencies.pl

+			catch {
+				print "Warning : Download failed for $fn from custom url $url: $_";
+			};
+			if (!$download_success && $url ne "") {


Why we need to check $url here?

llxia · 2025-10-29T21:00:05Z

scripts/getDependencies.pl

 		my $full_dir_path = File::Spec->catdir($path, $dir);
 		my $url_custom = $customUrl;
-
+		my $thirdParty_Url = $url;


You can use camel case or _, but do not mix them. In this case, call it third_party_url

llxia · 2025-10-29T21:00:59Z

scripts/getDependencies.pl

+					$download_success = 1;
+					}
+				catch {
+					print "Error: Failed to download $fn from third-party URL ($thirdParty_Url): $_";


If the second download failed, the program should exit.

llxia · 2025-10-29T21:03:09Z

scripts/getDependencies.pl

+			unless ($download_success) {
+				print " ERROR: Could not download $fn from either custom or third-party URL.\n";
+				next;
+			}


This can be removed.

llxia · 2025-10-29T21:08:37Z

Please also run extended.functional. Since we are using a new library Try::Tiny, please run sanity.functional on zos.

Amrutha-Kanhirathingal · 2025-10-30T11:05:50Z

Extended.functional : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55868/
Sanity.functional on ZOS : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55895/console : failed since zos don't have Try::Tiny module
hence updated script by using "eval" instead of "try-catch"

Logs :
sanity.functional -zos : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55914/console
Extended functional : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55918/.
special function : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55919/
Extended perf : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55920
sanity.perf : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55921/
extended perf ( failed - case): https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55909/
system : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/55923/consoleFull

llxia · 2025-10-30T17:25:29Z

scripts/getDependencies.pl

+				$download_success = 1;
+			};
+			if (!$download_success) {
+				print "Warning: Download failed for $fn from custom URL $url\nDownloading $fn from third-party URL: $third_party_url\n";


Please use $filename, not $fn

llxia · 2025-10-30T17:25:39Z

scripts/getDependencies.pl

+					$download_success = 1;
+				};
+				if (!$download_success) {
+					print "Error: Failed to download $fn from third-party URL $third_party_url\n";


Please use $filename, not $fn

LongyuZhang · 2025-10-30T17:46:54Z

scripts/getDependencies.pl

 use File::Copy;
 use File::Spec;
 use File::Path qw(make_path);
-


We can keep this blank line.

Deleted blank lines added back

llxia · 2025-10-30T17:57:32Z

Could you fix Closes : #6461 in both description and commit message? It should be using the full linkCloses: https://github.com/adoptium/aqa-tests/issues/6461

- getDependencies.pl script run twice.Once with customUrl. If it failed, retry with 3rd party URL. Closes: adoptium/aqa-tests#6461 Signed-off-by:Amrutha Kanhirathingal <Amrutha.Kanhirathingal@ibm.com>

LongyuZhang

LGTM

llxia · 2025-10-30T19:06:23Z

tested in parallel: extended.functional and sanity.functional

@Amrutha-Kanhirathingal, please fix failed cases in sanity.functional.

Amrutha-Kanhirathingal · 2025-10-31T07:28:21Z

@llxia
Issue fixed : eclipse-openj9/openj9#22877

llxia · 2025-12-01T16:20:25Z

This PR should run on the master branch (not the release branch) to ensure all dependent tests are updated accordingly.
@smlambert, I will leave it for you to merge.

annaibm · 2025-12-03T14:15:52Z

@Amrutha-Kanhirathingal,
Could you please rerun the sanity.functional and sanity.system tests to confirm everything is still working? Since a month has passed, I just want to make sure nothing has broken..

sophia-guo · 2025-12-04T22:24:30Z

scripts/getDependencies.pl


 # define directory path separator
 my $sep = File::Spec->catfile('', '');
+$customUrl = $ENV{'CUSTOM_URL'} if !defined($customUrl) || $customUrl eq '';


New environment variable means local usr ( not jenkins story) has to do another export. Instead of setting in Jenkins file would it be better to set in makefile ie, evnSettings.mk or settings.mk?

Yes Thanks for pointing this out.
https://github.com/adoptium/aqa-tests/blob/80a0985b5df03cbd05cebe9ae4a81a6a42bb8dfd/buildenv/jenkins/JenkinsfileBase#L215-L216
This is how we are fetching and assigning customUrl.
I’ll set the defaults in the Makefile as suggested.

Amrutha-Kanhirathingal · 2025-12-09T17:43:31Z

@Amrutha-Kanhirathingal, Could you please rerun the sanity.functional and sanity.system tests to confirm everything is still working? Since a month has passed, I just want to make sure nothing has broken..

sanity.functional : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/56773/console
sanity.system : https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/56781/

Amrutha-Kanhirathingal marked this pull request as draft October 24, 2025 14:43

Amrutha-Kanhirathingal force-pushed the TKG-TEST-PR branch 2 times, most recently from 0f6efab to f6657cc Compare October 28, 2025 18:34

Amrutha-Kanhirathingal marked this pull request as ready for review October 29, 2025 17:06

llxia requested a review from LongyuZhang October 29, 2025 20:50

llxia reviewed Oct 29, 2025

View reviewed changes

Amrutha-Kanhirathingal force-pushed the TKG-TEST-PR branch from f6657cc to 672883a Compare October 30, 2025 04:55

Amrutha-Kanhirathingal force-pushed the TKG-TEST-PR branch 2 times, most recently from dac6954 to 1163234 Compare October 30, 2025 16:33

llxia reviewed Oct 30, 2025

View reviewed changes

Amrutha-Kanhirathingal force-pushed the TKG-TEST-PR branch from 1163234 to 214c065 Compare October 30, 2025 17:31

LongyuZhang reviewed Oct 30, 2025

View reviewed changes

Amrutha-Kanhirathingal force-pushed the TKG-TEST-PR branch from 214c065 to c8dde07 Compare October 30, 2025 17:53

llxia mentioned this pull request Oct 30, 2025

Improved getDependency workflow adoptium/aqa-tests#6694

Open

Improved getDependency workflow

3833722

- getDependencies.pl script run twice.Once with customUrl. If it failed, retry with 3rd party URL. Closes: adoptium/aqa-tests#6461 Signed-off-by:Amrutha Kanhirathingal <Amrutha.Kanhirathingal@ibm.com>

Amrutha-Kanhirathingal force-pushed the TKG-TEST-PR branch from c8dde07 to 3833722 Compare October 30, 2025 18:00

LongyuZhang approved these changes Oct 30, 2025

View reviewed changes

llxia mentioned this pull request Nov 3, 2025

Add retry to getDependencies downloadFile #766

Merged

sxa mentioned this pull request Nov 4, 2025

Clear externalDependency and testDependency cached files periodically adoptium/infrastructure#4123

Open

llxia approved these changes Dec 1, 2025

View reviewed changes

sophia-guo reviewed Dec 4, 2025

View reviewed changes

Uh oh!

Improved getDependency workflow #758

Are you sure you want to change the base?

Improved getDependency workflow #758

Uh oh!

Conversation

Amrutha-Kanhirathingal commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Amrutha-Kanhirathingal commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llxia commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Amrutha-Kanhirathingal commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

llxia commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Amrutha-Kanhirathingal commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

llxia commented Oct 30, 2025

Uh oh!

LongyuZhang left a comment

Choose a reason for hiding this comment

Uh oh!

llxia commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Amrutha-Kanhirathingal commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llxia commented Dec 1, 2025

Uh oh!

annaibm commented Dec 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Amrutha-Kanhirathingal commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Amrutha-Kanhirathingal commented Oct 24, 2025 •

edited

Loading

Amrutha-Kanhirathingal commented Oct 24, 2025 •

edited

Loading

llxia commented Oct 24, 2025 •

edited

Loading

llxia commented Oct 29, 2025 •

edited

Loading

Amrutha-Kanhirathingal commented Oct 30, 2025 •

edited

Loading

llxia commented Oct 30, 2025 •

edited

Loading

Amrutha-Kanhirathingal commented Oct 31, 2025 •

edited

Loading