Skip to content

Data management and Globus pages reorganization #1115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bdu-birhanu wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Data management and Globus pages reorganization #1115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d6388b4
move storage related contents to storage category
bdu-birhanu Nov 26, 2025
aa98772
reorganize globus
bdu-birhanu Dec 5, 2025
aaed905
moving contents from group organization tutorial to gcp instal and u…
bdu-birhanu Dec 5, 2025
99382cf
adding index pages for DM
bdu-birhanu Dec 8, 2025
e9d2000
adding short table of content to index DM index page
bdu-birhanu Dec 12, 2025
6e823f9
remove gpc installation section from the tutorial
bdu-birhanu Dec 17, 2025
7d5433e
move files from tutorial directory to globus directory
bdu-birhanu Jan 5, 2026
fa96e57
move all image from tutorial/images directory to globus/images
bdu-birhanu Jan 5, 2026
08d432d
Remove GCP page and images under tutorial directory
bdu-birhanu Jan 5, 2026
b8ddd0c
Update LTS pages
bdu-birhanu Jan 5, 2026
dd202ee
Update the main data mgt index page and delete tutorial folder
bdu-birhanu Jan 5, 2026
057c633
remove globus_organization_tutorial from tutotial directory
bdu-birhanu Jan 5, 2026
96a5fdd
update a globus individaul tutorial page, and related cross-links
bdu-birhanu Jan 9, 2026
e63affc
update screenshoots in globus for individual researcher page
bdu-birhanu Jan 13, 2026
cd68f0f
Add the new globus collection name to globus_organization page
bdu-birhanu Jan 15, 2026
8eb9551
Add cross-links and update nav for the globus tutorial
bdu-birhanu Jan 16, 2026
dc84a22
Merge branch 'main' into feat-globus-reorg
bdu-birhanu Jan 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/account_management/cheaha_account.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,10 +100,10 @@ We're always happy to provide support for your Research Computing needs, you nee

### For Lab PIs and Core Directors

- [No-cost storage offerings](../data_management/index.md#what-type-of-storage-do-i-need)
- [GPFS](../data_management/index.md#what-shared-storage-solutions-are-available): Hot storage, compute adjacent, directly accessible from Cheaha
- [LTS](../data_management/lts/index.md): Cool storage, large capacity
- [Transfer data with Globus](../data_management/transfer/globus.md)
- [No-cost storage offerings](../data_management/storage/index.md#what-type-of-storage-do-i-need)
- [GPFS](../data_management/storage/index.md#what-shared-storage-solutions-are-available): Hot storage, compute adjacent, directly accessible from Cheaha
- [LTS](../data_management/storage/lts/index.md): Cool storage, large capacity
- [Transfer data with Globus](../data_management/transfer/globus/index.md)
- [Batch computing](../cheaha/slurm/introduction.md)
- Interactive applications in [Open OnDemand](../cheaha/open_ondemand/index.md)
- [HPC Desktop](../cheaha/open_ondemand/hpc_desktop.md)
Expand Down
8 changes: 4 additions & 4 deletions docs/account_management/leaving_uab.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ For each dataset or project, which option is ideal depends on the state of the p
- Transfer responsibility of the data to a successor (retiring)?

1. Make the appropriate arrangements with relevant administrative units (e.g., Office of Research and OSP).
1. [Globus](../data_management/transfer/globus.md) may be used to efficiently and robustly transfer the data from one directory on Cheaha to another. Please [contact us](../help/support.md) for more details.
1. [Globus](../data_management/transfer/globus/index.md) may be used to efficiently and robustly transfer the data from one directory on Cheaha to another. Please [contact us](../help/support.md) for more details.

- Transfer the data to a new instutition (moving)?

Expand All @@ -70,9 +70,9 @@ Cores should not assume responsibility for any of the research data they produce

If the Core will continue operating, then it is assumed there will be a successor. Take whatever necessary administrative steps to transfer control and responsibility for Core operations. Things that may need to be transfered:

- [GPFS (Cheaha) shared allocation](../data_management/cheaha_storage_gpfs/project_directories.md).
- [LTS shared allocation](../data_management/lts/index.md).
- [GPFS (Cheaha) shared allocation](../data_management/storage/cheaha_storage_gpfs/project_directories.md).
- [LTS shared allocation](../data_management/storage/lts/index.md).
- [OpenStack Shared Projects](../uab_cloud/sharing_cloud_environment.md).
- Service account, shared mailboxes, and other shared resources not managed by Research Computing, but which may have an impact on Core operations relating to Research Data acquisition and transfer, such as [Globus](../data_management/transfer/globus.md).
- Service account, shared mailboxes, and other shared resources not managed by Research Computing, but which may have an impact on Core operations relating to Research Data acquisition and transfer, such as [Globus](../data_management/transfer/globus/index.md).

If the Core will cease operating, please [contact us](../help/support.md) to make arrangements to ensure final disposition of all research data.
2 changes: 1 addition & 1 deletion docs/cheaha/getting_started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ All researchers are granted 5 TB of individual storage when they [create their R

Shared storage is available to all Lab Groups and Core Facilities on campus. Shared storage is also available to UAB Administration groups.

Please visit our [Storage page](../data_management/index.md) for detailed information about our individual and shared storage options.
Please visit our [Storage page](../data_management/storage/index.md) for detailed information about our individual and shared storage options.

### Partitions

Expand Down
2 changes: 1 addition & 1 deletion docs/cheaha/job_efficiency.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Questions to ask yourself before requesting resources:
1. How is the software I'm using programmed?

- Can it use a GPU? Request one. Don't forget to consider...
- [Local Scratch](../data_management/cheaha_storage_gpfs/local_scratch.md) for [IO performance](../cheaha/slurm/gpu.md#ensuring-io-performance-with-a100-gpus).
- [Local Scratch](../data_management/storage/cheaha_storage_gpfs/local_scratch.md) for [IO performance](../cheaha/slurm/gpu.md#ensuring-io-performance-with-a100-gpus).
- `--ntasks-per-socket` when using [Multiple GPUs](../cheaha/slurm/gpu.md#using-multiple-gpus).
- Can it use multiple cores? Request more than one core.
- Is it single-threaded? Request only one core.
Expand Down
2 changes: 1 addition & 1 deletion docs/cheaha/open_ondemand/hpc_desktop.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Once you click the tab, you'll see the control panel appear. The second option f

![!VNC clipboard](images/ood_desktop_copy_paste.png)

To copy from the VNC to your personal machine, highlight the text you want to copy in the VNC session, and that text will appear in the clipboard. Select the text in the clipboard, copy it, and then paste it on your local machine. Images cannot be copy-pasted through this clipboard. Instead, images should be saved as a file and then transferred through tools such as [Globus](../../data_management/transfer/globus.md), [rclone](../../data_management/transfer/rclone.md), or an [scp utility](https://servicenow.iu.edu/kb?id=kb_article_view&sysparm_article=KB0024361).
To copy from the VNC to your personal machine, highlight the text you want to copy in the VNC session, and that text will appear in the clipboard. Select the text in the clipboard, copy it, and then paste it on your local machine. Images cannot be copy-pasted through this clipboard. Instead, images should be saved as a file and then transferred through tools such as [Globus](../../data_management/transfer/globus/index.md), [rclone](../../data_management/transfer/rclone.md), or an [scp utility](https://servicenow.iu.edu/kb?id=kb_article_view&sysparm_article=KB0024361).

## Visual Studio Code Remote Tunnel

Expand Down
10 changes: 5 additions & 5 deletions docs/cheaha/open_ondemand/ood_layout.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ The landing page contains information about current updates to the system, a mes

You can open a file browser in a new tab by clicking the `Files` dropdown and selecting which default directory you would like to access from the following options.

- [Home Directory](../../data_management/cheaha_storage_gpfs/individual_directories.md)
- [/data/scratch/$USER](../../data_management/cheaha_storage_gpfs/network_scratch.md) (network scratch)
- [/data/user/$USER](../../data_management/cheaha_storage_gpfs/individual_directories.md)
- [/scratch/$USER](../../data_management/cheaha_storage_gpfs/network_scratch.md) (network scratch)
- [Home Directory](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md)
- [/data/scratch/$USER](../../data_management/storage/cheaha_storage_gpfs/network_scratch.md) (network scratch)
- [/data/user/$USER](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md)
- [/scratch/$USER](../../data_management/storage/cheaha_storage_gpfs/network_scratch.md) (network scratch)

<!-- markdownlint-disable MD046 -->
!!! note
Expand Down Expand Up @@ -52,7 +52,7 @@ Data can be uploaded from your local machine using this interface. Use the `Uplo
<!-- markdownlint-disable MD046 -->
!!! important

This should be limited to small files only (< 1 MB). For large files or datasets, please use [Globus](../../data_management/transfer/globus.md) instead.
This should be limited to small files only (< 1 MB). For large files or datasets, please use [Globus](../../data_management/transfer/globus/index.md) instead.
<!-- markdownlint-enable MD046 -->

#### Opening a Terminal
Expand Down
4 changes: 2 additions & 2 deletions docs/cheaha/slurm/gpu.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ If you are running a GPU-enabled software inside a [Singularity container](../..

#### Ensuring IO Performance With A100 GPUs

If you are using `amperenodes` and the A100 GPUs, then we strongly recommend moving your input files to [local scratch](../../data_management/cheaha_storage_gpfs/local_scratch.md). Local scratch on the A100 nodes are fast, striped solid state drives (SSDs) and are have enough throughput to keep the A100 GPUs busy. Network storage systems, including [network scratch](../../data_management/cheaha_storage_gpfs/network_scratch.md), [project directories](../../data_management/cheaha_storage_gpfs/project_directories.md), and [individual directories](../../data_management/cheaha_storage_gpfs/individual_directories.md), have lower throughput. To optimize A100 efficiency, use local scratch.
If you are using `amperenodes` and the A100 GPUs, then we strongly recommend moving your input files to [local scratch](../../data_management/storage/cheaha_storage_gpfs/local_scratch.md). Local scratch on the A100 nodes are fast, striped solid state drives (SSDs) and are have enough throughput to keep the A100 GPUs busy. Network storage systems, including [network scratch](../../data_management/storage/cheaha_storage_gpfs/network_scratch.md), [project directories](../../data_management/storage/cheaha_storage_gpfs/project_directories.md), and [individual directories](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md), have lower throughput. To optimize A100 efficiency, use local scratch.

Please see our [Local Scratch page](../../data_management/cheaha_storage_gpfs/local_scratch.md) for more details and a template script.
Please see our [Local Scratch page](../../data_management/storage/cheaha_storage_gpfs/local_scratch.md) for more details and a template script.

#### Using Multiple GPUs

Expand Down
2 changes: 1 addition & 1 deletion docs/cheaha/slurm/introduction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ The official Slurm documentation can be found at [the Slurm website](https://slu
## How Do I Use Slurm?

1. Understand how to run your software in 'batch' mode, purely from the command line. You may need to review the software documentation.
1. Stage data to [Cheaha storage](../../data_management/cheaha_storage_gpfs/index.md).
1. Stage data to [Cheaha storage](../../data_management/storage/cheaha_storage_gpfs/index.md).
1. Figure out what resources are needed, such as CPUs, memory, GPUs, and time.
1. Write a [Slurm job script](./slurm_tutorial.md). Be sure to delete data from local scratch.
1. [Submit the job](./submitting_jobs.md) with the `sbatch` command.
Expand Down
2 changes: 1 addition & 1 deletion docs/cheaha/slurm/slurm_tutorial.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -712,7 +712,7 @@ In this section there are two tutorials that show how to use (a) a single GPU, a
<!-- markdownlint-disable MD046 -->
!!! note

For real applications, especially AI and other large-data applications, we recommend pre-loading data onto [Local Scratch](../../data_management/cheaha_storage_gpfs/local_scratch.md) to [ensure good performance](../slurm/gpu.md#ensuring-io-performance-with-a100-gpus). Don't worry about doing this for the current tutorial, but do make a note of it for your own scientific work. The difference in performance is huge, especially for AI and large-data applications.
For real applications, especially AI and other large-data applications, we recommend pre-loading data onto [Local Scratch](../../data_management/storage/cheaha_storage_gpfs/local_scratch.md) to [ensure good performance](../slurm/gpu.md#ensuring-io-performance-with-a100-gpus). Don't worry about doing this for the current tutorial, but do make a note of it for your own scientific work. The difference in performance is huge, especially for AI and large-data applications.
<!-- markdownlint-enable MD046 -->

#### Initial Setup
Expand Down
4 changes: 2 additions & 2 deletions docs/cheaha/slurm/submitting_jobs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ For batch jobs, flags are typically included as directive comments at the top of

### A Simple Batch Job

Below is an example batch job script. To test it, copy and paste it into a plain text file `testjob.sh` in your [Home Directory](../../data_management/cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) on Cheaha. Run it at the terminal by navigating to your home directory by entering `cd ~` and then entering `sbatch testjob.sh`. Momentarily, two text files with `.out` and `.err` suffixes will be produced in your home directory.
Below is an example batch job script. To test it, copy and paste it into a plain text file `testjob.sh` in your [Home Directory](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) on Cheaha. Run it at the terminal by navigating to your home directory by entering `cd ~` and then entering `sbatch testjob.sh`. Momentarily, two text files with `.out` and `.err` suffixes will be produced in your home directory.

```bash linenums="1"
#!/bin/bash
Expand Down Expand Up @@ -102,7 +102,7 @@ There is a lot going on in the above script, so let's break it down. There are t

Building on the job script above, below is an array job. Array jobs are useful when you need to perform the same analysis on slightly different inputs with no interaction between those analyses. We call this situation "pleasingly parallel". We can take advantage of an array job using the variable `$SLURM_ARRAY_TASK_ID`, which will have an integer in the set of values we give to the `--array` flag.

To test the script below, copy and paste it into a plain text file `testarrayjob.sh` in your [Home Directory](../../data_management/cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) on Cheaha. Run it at the terminal by navigating to your home directory by entering `cd ~` and then entering `sbatch testarrayjob.sh`. Momentarily, 16 text files with `.out` and `.err` suffixes will be produced in your home directory.
To test the script below, copy and paste it into a plain text file `testarrayjob.sh` in your [Home Directory](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) on Cheaha. Run it at the terminal by navigating to your home directory by entering `cd ~` and then entering `sbatch testarrayjob.sh`. Momentarily, 16 text files with `.out` and `.err` suffixes will be produced in your home directory.

```bash linenums="1"
#!/bin/bash
Expand Down
Loading
Loading