Add Internal Benchmark tooling

[wildintellect]

To facilitate tracking of benchmarks internal to the job runs the MAAP team is recommending Scalene for python code.

• Wrap existing algorithm in Scalene

[chuckwondo]

I implemented this on a new branch: https://github.com/MAAP-Project/get-dem/tree/scalene

I deployed a new version of the algorithm as GET-DEM:scalene and ran it with the following inputs:

• bbox: -156.0 18.8 -154.7 20.3
• compute: true
• scalene_args: (left empty to use defaults)

I copied the outputs to /projects/shared-buckets/dschuck/get-dem/

I also tested that I can run it successfully from the ADE.

[chuckwondo]

Closed too soon. Need PR approved first.

[chuckwondo]

@nemo794, I made adjustments to produce a JSON profile by default, rather than HTML. (I'm continuing to make adjustments to the scalene branch, which we can pick apart for submitting incremental PRs once #2 lands.)

I also modified the notebook in a manner that should aid our eventual effort to collate/aggregate profiling metrics once we're ready to kick off a slew of jobs. The primary change was to associate names with the sample bboxes to allow us to use the names as part of the tags for the jobs we submit, because a job's tag is used to create a directory along the path structure to the job's output directory.

More specifically, in the notebook, a job's tag (the identifier argument to submitJob, but labeled "Tag" in the Jobs UI) is structured as {bbox_name}__compute__{queue} when compute is True, or {bbox_name}__no-compute__{queue} when compute is False.

As an example, I ran "Italy", and I copied the output to /projects/shared-buckets/dschuck/get-dem/scalene/Italy__compute__maap-dps-worker-32vcpu-64gb/2024/03/08/07/57/58/466407/ so you can see the results.

Note that the path from get-dem onward is the full structure I copied from /projects/my-private-bucket/dps_output. Also note that the directory Italy__compute__maap-dps-worker-32vcpu-64gb is the job tag.

The information we want to extract from profile.json is within the files/<python filename>/functions arrays (only 1 python file in our case).

Unfortunately, Scalene does not produce an HTML file corresponding to the JSON file, so it's not immediately and completely clear which stats from the JSON file are the ones that are nicely rendered in the HTML file, so my next step is to run another job with the same inputs, but passing arguments to scalene to produce an HTML file. Then I'll compare what I see in the browser for the HTML profile against the stats in the JSON file to match things up. (Obviously, the numbers will be different, but should be close enough for me to decipher things.)

[chuckwondo]

@nemo794, it looks like these are the 2 primary values we want to pull out of the JSON profile:

• elapsed_time_sec (time)
• max_footprint_mb (space)

What other information might be needed for the large spreadsheet you presented in the recent meeting about gathering get-dem performance figures from both NASA and ESA jobs?

[chuckwondo]

@nemo794, for comparison between the 2 Italy runs (one producing profile.json and the other profile.html), I've copied the second run to /projects/shared-buckets/dschuck/get-dem/scalene/Italy__compute__maap-dps-worker-32vcpu-64gb/2024/03/08/10/03/45/609726/

[nemo794]

Hi @chuckwondo , This is a great progress! Thank you! We discussed this offline, but for the record, I really like how you modularized each algorithm step into its own function, in order to wrap each step individually in the scalene wrapper.

The DPS outputs are really helpful! A few thoughts:

• Is Scalene able to generate both html and JSON outputs at the same time? JSON has higher priority because of the need to parse algorithmically, but as a human the html is much friendlier for spot checking.
• max_footprint_mb (space)
  • Good call -- this overview metric is helpful. If the number of cores remains constant, I don't see any reason this value should change from run to run... but it will help to select correct instance types, see impacts from changing the number of cores, and also as a sanity check that the algorithm isn't doing anything fishy. For now, I don't think we need a finer level of granularity.
• elapsed_time_sec (time)
  • This appears to be the total time of the algorithm. This is definitely a good metric to have; the MAAP platforms' DPS metrics should actually capture the same metric, so now we'll be able to compare that the platforms are correctly capturing that info. Nice!
  • For benchmarking, we'll need a finer level of granularity. It'd be great to have timings for each step. Looking at the JSON, I'm only seeing the percentages of time and total time, but maybe I'm missing something? Here are a few screenshots to help clarify:
    
    

After this, hopefully there's a way to get even finer granularity by digging into the sardem step via calling it directly from Python, but one step at a time. :)

[chuckwondo]

Hi @nemo794, thanks for the great annotated screenshots. They are very helpful.

This perhaps got buried in one of my earlier comments:

Unfortunately, Scalene does not produce an HTML file corresponding to the JSON file

That is, Scalene won't produce both formats for a given run. It's one or the other, sadly.

Regarding, "finer granularity by digging into the sardem step via calling it directly from Python," I forgot to mention that I've already done that. The following shows the replacement of the system call:

That's within the get_dem function, so now we are getting the mem+cpu+io information that we were not seeing when we were using the system call.

Regarding the absolute time values vs. the percentages, it seems strange that the profile does not contain the individual elapsed time values. Unfortunately, I think that leaves us with having to do the extra calculations ourselves, but at least we have the necessary numbers to do so.

[chuckwondo]

Fixed by #6.

[nemo794]

Let's keep this open until the json parser is integrated. Thanks!