diff --git a/docs/account_management/cheaha_account.md b/docs/account_management/cheaha_account.md index 7d4a407ca..f334718da 100644 --- a/docs/account_management/cheaha_account.md +++ b/docs/account_management/cheaha_account.md @@ -100,10 +100,10 @@ We're always happy to provide support for your Research Computing needs, you nee ### For Lab PIs and Core Directors -- [No-cost storage offerings](../data_management/index.md#what-type-of-storage-do-i-need) - - [GPFS](../data_management/index.md#what-shared-storage-solutions-are-available): Hot storage, compute adjacent, directly accessible from Cheaha - - [LTS](../data_management/lts/index.md): Cool storage, large capacity - - [Transfer data with Globus](../data_management/transfer/globus.md) +- [No-cost storage offerings](../data_management/storage/index.md#what-type-of-storage-do-i-need) + - [GPFS](../data_management/storage/index.md#what-shared-storage-solutions-are-available): Hot storage, compute adjacent, directly accessible from Cheaha + - [LTS](../data_management/storage/lts/index.md): Cool storage, large capacity + - [Transfer data with Globus](../data_management/transfer/globus/index.md) - [Batch computing](../cheaha/slurm/introduction.md) - Interactive applications in [Open OnDemand](../cheaha/open_ondemand/index.md) - [HPC Desktop](../cheaha/open_ondemand/hpc_desktop.md) diff --git a/docs/account_management/leaving_uab.md b/docs/account_management/leaving_uab.md index 970a49a35..1b2420493 100644 --- a/docs/account_management/leaving_uab.md +++ b/docs/account_management/leaving_uab.md @@ -50,7 +50,7 @@ For each dataset or project, which option is ideal depends on the state of the p - Transfer responsibility of the data to a successor (retiring)? 1. Make the appropriate arrangements with relevant administrative units (e.g., Office of Research and OSP). - 1. [Globus](../data_management/transfer/globus.md) may be used to efficiently and robustly transfer the data from one directory on Cheaha to another. Please [contact us](../help/support.md) for more details. + 1. [Globus](../data_management/transfer/globus/index.md) may be used to efficiently and robustly transfer the data from one directory on Cheaha to another. Please [contact us](../help/support.md) for more details. - Transfer the data to a new instutition (moving)? @@ -70,9 +70,9 @@ Cores should not assume responsibility for any of the research data they produce If the Core will continue operating, then it is assumed there will be a successor. Take whatever necessary administrative steps to transfer control and responsibility for Core operations. Things that may need to be transfered: -- [GPFS (Cheaha) shared allocation](../data_management/cheaha_storage_gpfs/project_directories.md). -- [LTS shared allocation](../data_management/lts/index.md). +- [GPFS (Cheaha) shared allocation](../data_management/storage/cheaha_storage_gpfs/project_directories.md). +- [LTS shared allocation](../data_management/storage/lts/index.md). - [OpenStack Shared Projects](../uab_cloud/sharing_cloud_environment.md). -- Service account, shared mailboxes, and other shared resources not managed by Research Computing, but which may have an impact on Core operations relating to Research Data acquisition and transfer, such as [Globus](../data_management/transfer/globus.md). +- Service account, shared mailboxes, and other shared resources not managed by Research Computing, but which may have an impact on Core operations relating to Research Data acquisition and transfer, such as [Globus](../data_management/transfer/globus/index.md). If the Core will cease operating, please [contact us](../help/support.md) to make arrangements to ensure final disposition of all research data. diff --git a/docs/cheaha/getting_started.md b/docs/cheaha/getting_started.md index d5cfbe8e7..8aaee7172 100644 --- a/docs/cheaha/getting_started.md +++ b/docs/cheaha/getting_started.md @@ -53,7 +53,7 @@ All researchers are granted 5 TB of individual storage when they [create their R Shared storage is available to all Lab Groups and Core Facilities on campus. Shared storage is also available to UAB Administration groups. -Please visit our [Storage page](../data_management/index.md) for detailed information about our individual and shared storage options. +Please visit our [Storage page](../data_management/storage/index.md) for detailed information about our individual and shared storage options. ### Partitions diff --git a/docs/cheaha/job_efficiency.md b/docs/cheaha/job_efficiency.md index 658f368f4..2b0b7b3db 100644 --- a/docs/cheaha/job_efficiency.md +++ b/docs/cheaha/job_efficiency.md @@ -42,7 +42,7 @@ Questions to ask yourself before requesting resources: 1. How is the software I'm using programmed? - Can it use a GPU? Request one. Don't forget to consider... - - [Local Scratch](../data_management/cheaha_storage_gpfs/local_scratch.md) for [IO performance](../cheaha/slurm/gpu.md#ensuring-io-performance-with-a100-gpus). + - [Local Scratch](../data_management/storage/cheaha_storage_gpfs/local_scratch.md) for [IO performance](../cheaha/slurm/gpu.md#ensuring-io-performance-with-a100-gpus). - `--ntasks-per-socket` when using [Multiple GPUs](../cheaha/slurm/gpu.md#using-multiple-gpus). - Can it use multiple cores? Request more than one core. - Is it single-threaded? Request only one core. diff --git a/docs/cheaha/open_ondemand/hpc_desktop.md b/docs/cheaha/open_ondemand/hpc_desktop.md index 18fb7bc81..41fba6e62 100644 --- a/docs/cheaha/open_ondemand/hpc_desktop.md +++ b/docs/cheaha/open_ondemand/hpc_desktop.md @@ -74,7 +74,7 @@ Once you click the tab, you'll see the control panel appear. The second option f ![!VNC clipboard](images/ood_desktop_copy_paste.png) -To copy from the VNC to your personal machine, highlight the text you want to copy in the VNC session, and that text will appear in the clipboard. Select the text in the clipboard, copy it, and then paste it on your local machine. Images cannot be copy-pasted through this clipboard. Instead, images should be saved as a file and then transferred through tools such as [Globus](../../data_management/transfer/globus.md), [rclone](../../data_management/transfer/rclone.md), or an [scp utility](https://servicenow.iu.edu/kb?id=kb_article_view&sysparm_article=KB0024361). +To copy from the VNC to your personal machine, highlight the text you want to copy in the VNC session, and that text will appear in the clipboard. Select the text in the clipboard, copy it, and then paste it on your local machine. Images cannot be copy-pasted through this clipboard. Instead, images should be saved as a file and then transferred through tools such as [Globus](../../data_management/transfer/globus/index.md), [rclone](../../data_management/transfer/rclone.md), or an [scp utility](https://servicenow.iu.edu/kb?id=kb_article_view&sysparm_article=KB0024361). ## Visual Studio Code Remote Tunnel diff --git a/docs/cheaha/open_ondemand/ood_layout.md b/docs/cheaha/open_ondemand/ood_layout.md index 81430c25e..2a90e9a3b 100644 --- a/docs/cheaha/open_ondemand/ood_layout.md +++ b/docs/cheaha/open_ondemand/ood_layout.md @@ -10,10 +10,10 @@ The landing page contains information about current updates to the system, a mes You can open a file browser in a new tab by clicking the `Files` dropdown and selecting which default directory you would like to access from the following options. -- [Home Directory](../../data_management/cheaha_storage_gpfs/individual_directories.md) -- [/data/scratch/$USER](../../data_management/cheaha_storage_gpfs/network_scratch.md) (network scratch) -- [/data/user/$USER](../../data_management/cheaha_storage_gpfs/individual_directories.md) -- [/scratch/$USER](../../data_management/cheaha_storage_gpfs/network_scratch.md) (network scratch) +- [Home Directory](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md) +- [/data/scratch/$USER](../../data_management/storage/cheaha_storage_gpfs/network_scratch.md) (network scratch) +- [/data/user/$USER](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md) +- [/scratch/$USER](../../data_management/storage/cheaha_storage_gpfs/network_scratch.md) (network scratch) !!! note @@ -52,7 +52,7 @@ Data can be uploaded from your local machine using this interface. Use the `Uplo !!! important - This should be limited to small files only (< 1 MB). For large files or datasets, please use [Globus](../../data_management/transfer/globus.md) instead. + This should be limited to small files only (< 1 MB). For large files or datasets, please use [Globus](../../data_management/transfer/globus/index.md) instead. #### Opening a Terminal diff --git a/docs/cheaha/slurm/gpu.md b/docs/cheaha/slurm/gpu.md index 512d19f3f..50b750c42 100644 --- a/docs/cheaha/slurm/gpu.md +++ b/docs/cheaha/slurm/gpu.md @@ -29,9 +29,9 @@ If you are running a GPU-enabled software inside a [Singularity container](../.. #### Ensuring IO Performance With A100 GPUs -If you are using `amperenodes` and the A100 GPUs, then we strongly recommend moving your input files to [local scratch](../../data_management/cheaha_storage_gpfs/local_scratch.md). Local scratch on the A100 nodes are fast, striped solid state drives (SSDs) and are have enough throughput to keep the A100 GPUs busy. Network storage systems, including [network scratch](../../data_management/cheaha_storage_gpfs/network_scratch.md), [project directories](../../data_management/cheaha_storage_gpfs/project_directories.md), and [individual directories](../../data_management/cheaha_storage_gpfs/individual_directories.md), have lower throughput. To optimize A100 efficiency, use local scratch. +If you are using `amperenodes` and the A100 GPUs, then we strongly recommend moving your input files to [local scratch](../../data_management/storage/cheaha_storage_gpfs/local_scratch.md). Local scratch on the A100 nodes are fast, striped solid state drives (SSDs) and are have enough throughput to keep the A100 GPUs busy. Network storage systems, including [network scratch](../../data_management/storage/cheaha_storage_gpfs/network_scratch.md), [project directories](../../data_management/storage/cheaha_storage_gpfs/project_directories.md), and [individual directories](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md), have lower throughput. To optimize A100 efficiency, use local scratch. -Please see our [Local Scratch page](../../data_management/cheaha_storage_gpfs/local_scratch.md) for more details and a template script. +Please see our [Local Scratch page](../../data_management/storage/cheaha_storage_gpfs/local_scratch.md) for more details and a template script. #### Using Multiple GPUs diff --git a/docs/cheaha/slurm/introduction.md b/docs/cheaha/slurm/introduction.md index 6e5eb5547..7fb10cfbb 100644 --- a/docs/cheaha/slurm/introduction.md +++ b/docs/cheaha/slurm/introduction.md @@ -7,7 +7,7 @@ The official Slurm documentation can be found at [the Slurm website](https://slu ## How Do I Use Slurm? 1. Understand how to run your software in 'batch' mode, purely from the command line. You may need to review the software documentation. -1. Stage data to [Cheaha storage](../../data_management/cheaha_storage_gpfs/index.md). +1. Stage data to [Cheaha storage](../../data_management/storage/cheaha_storage_gpfs/index.md). 1. Figure out what resources are needed, such as CPUs, memory, GPUs, and time. 1. Write a [Slurm job script](./slurm_tutorial.md). Be sure to delete data from local scratch. 1. [Submit the job](./submitting_jobs.md) with the `sbatch` command. diff --git a/docs/cheaha/slurm/slurm_tutorial.md b/docs/cheaha/slurm/slurm_tutorial.md index 12f833f1e..3402dda15 100644 --- a/docs/cheaha/slurm/slurm_tutorial.md +++ b/docs/cheaha/slurm/slurm_tutorial.md @@ -712,7 +712,7 @@ In this section there are two tutorials that show how to use (a) a single GPU, a !!! note - For real applications, especially AI and other large-data applications, we recommend pre-loading data onto [Local Scratch](../../data_management/cheaha_storage_gpfs/local_scratch.md) to [ensure good performance](../slurm/gpu.md#ensuring-io-performance-with-a100-gpus). Don't worry about doing this for the current tutorial, but do make a note of it for your own scientific work. The difference in performance is huge, especially for AI and large-data applications. + For real applications, especially AI and other large-data applications, we recommend pre-loading data onto [Local Scratch](../../data_management/storage/cheaha_storage_gpfs/local_scratch.md) to [ensure good performance](../slurm/gpu.md#ensuring-io-performance-with-a100-gpus). Don't worry about doing this for the current tutorial, but do make a note of it for your own scientific work. The difference in performance is huge, especially for AI and large-data applications. #### Initial Setup diff --git a/docs/cheaha/slurm/submitting_jobs.md b/docs/cheaha/slurm/submitting_jobs.md index e469ee70a..9cfa64b89 100644 --- a/docs/cheaha/slurm/submitting_jobs.md +++ b/docs/cheaha/slurm/submitting_jobs.md @@ -65,7 +65,7 @@ For batch jobs, flags are typically included as directive comments at the top of ### A Simple Batch Job -Below is an example batch job script. To test it, copy and paste it into a plain text file `testjob.sh` in your [Home Directory](../../data_management/cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) on Cheaha. Run it at the terminal by navigating to your home directory by entering `cd ~` and then entering `sbatch testjob.sh`. Momentarily, two text files with `.out` and `.err` suffixes will be produced in your home directory. +Below is an example batch job script. To test it, copy and paste it into a plain text file `testjob.sh` in your [Home Directory](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) on Cheaha. Run it at the terminal by navigating to your home directory by entering `cd ~` and then entering `sbatch testjob.sh`. Momentarily, two text files with `.out` and `.err` suffixes will be produced in your home directory. ```bash linenums="1" #!/bin/bash @@ -102,7 +102,7 @@ There is a lot going on in the above script, so let's break it down. There are t Building on the job script above, below is an array job. Array jobs are useful when you need to perform the same analysis on slightly different inputs with no interaction between those analyses. We call this situation "pleasingly parallel". We can take advantage of an array job using the variable `$SLURM_ARRAY_TASK_ID`, which will have an integer in the set of values we give to the `--array` flag. -To test the script below, copy and paste it into a plain text file `testarrayjob.sh` in your [Home Directory](../../data_management/cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) on Cheaha. Run it at the terminal by navigating to your home directory by entering `cd ~` and then entering `sbatch testarrayjob.sh`. Momentarily, 16 text files with `.out` and `.err` suffixes will be produced in your home directory. +To test the script below, copy and paste it into a plain text file `testarrayjob.sh` in your [Home Directory](../../data_management/storage/cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) on Cheaha. Run it at the terminal by navigating to your home directory by entering `cd ~` and then entering `sbatch testarrayjob.sh`. Momentarily, 16 text files with `.out` and `.err` suffixes will be produced in your home directory. ```bash linenums="1" #!/bin/bash diff --git a/docs/data_management/index.md b/docs/data_management/index.md index f66b3a22f..1c88028c7 100644 --- a/docs/data_management/index.md +++ b/docs/data_management/index.md @@ -1,226 +1,15 @@ -# Storage +# Data Management -Research Computing offers several data storage options to meet individual or shared needs of UAB researchers, depending on their requirement and use-cases. The types of storage available, procedures for requesting access, responsibilities, and usage guidelines are detailed in the following sections. +Proper research data management ensures security, compliance, and operational efficiency, while ensuring that only authorized individuals have access to specific data. Researchers and users of Cheaha are responsible for organizing their data and backing up critical information. -## What Type of Storage Do I Need? +This page provides an overview of the storage options and data transfer tools available for research projects. It also highlights good practices for managing, sharing, and preserving research data, as well as the roles and responsibilities of researchers and research computing teams. -There are multiple locations for data storage both on and off Cheaha each with a specific purpose. You can look at the table below to help determine the storage platform we provide that best matches your needed use-case. After reviewing the table, proceed to the [How to Choose the Right Storage for My Use Case](#how-to-choose-the-right-storage-for-my-use-case) section for additional guidance on selecting the most suitable storage for your needs. If you need additional assistance, please contact [Support](../help/support.md). +We covered the following topics: -{{ read_csv('data_management/res/storage_overview.csv', keep_default_na=False) }} - -### How to Choose the Right Storage for My Use Case? - -When you have different types of data with varying storage needs, choosing the right storage is not an easy task. Below, we will walk you through common data storage needs and use cases. - -- **I have lots of data that rarely or never changes**: - - If your data is large and does not change frequently, [Long-Term Storage (LTS)](../data_management/lts/index.md) is your ideal option. -- **I need to do a lot of processing on the data**: - - If your data is frequently accessed or modified, [Cheaha Storage (GPFS)](../data_management/cheaha_storage_gpfs/index.md) is a perfect option for your use case. - - When you have finished working with the data, please consider putting it into [LTS](../data_management/lts/index.md). Please [Contact Us](../help/support.md#how-to-request-support) for more information. -- **I am just learning how to use Cheaha for very small personal projects**: - - If you are new to Cheaha and working with small personal projects, you do not need a complex storage solution. [User Data](../data_management/cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) is a simple and accessible option for beginners and smaller datasets. - - If you find your project growing, or you need to collaborate with others at UAB, consider a shared allocation on [GPFS](../data_management/cheaha_storage_gpfs/index.md) and/or [LTS](../data_management/lts/index.md). Please refer to the instructions on [how to request a shared allocation](#how-do-i-request-shared-storage). -- **I need somewhere to store lots of temporary files while I do my processing**: - - If you need a place to store large amounts of temporary data while processing data, [Network Scratch](../data_management/cheaha_storage_gpfs/network_scratch.md) is the best option. -- **I am working with high-throughput or I/O bound applications, like AI training or evaluation, or I have many very small files**: - - For high-performance tasks, like AI training, [Local Scratch](../data_management/cheaha_storage_gpfs/local_scratch.md) provides fast and optimized data access for intensive I/O workloads.. -- **I have a combination of needs and am unsure of how to proceed**: - - If your use case spans across multiple types of storage needs, [Contact Us](../help/support.md#how-to-request-support) to discuss your requirements, and we wll help you find the best solution. - -## What Individual Storage Solutions Are Available? - -Every Cheaha user has personal directories found at `/home/$USER` (or `$HOME`) and `/data/user/$USER` (or `$USER_DATA`), which are created automatically during account registration. In addition, individual allocations on Long-Term Storage (LTS) are also available upon request. Please read more about [Long-Term Storage](./lts/index.md) and [User Data and Home Directories](./cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories). - -### How Do I Request Individual Cheaha (`GPFS`) Storage? - -An individual Cheaha (`GPFS`) storage is automatically created as part of the Cheaha account setup. No separate request is required. - -### How Do I Request Individual Long-Term Storage? - -To request individual Long-Term Storage, please first read and understand how [Long-Term Storage](./lts/index.md) differs from traditional file systems, like GPFS on Cheaha. Decide if it is suitable for your needs. Then please feel free to contact [Support](../help/support.md). - -## What Shared Storage Solutions Are Available? - -Shared Storage is available via two services. We have [Project Storage](./cheaha_storage_gpfs/project_directories.md) (located in `/data/project` or Cheaha) and [Long-Term Storage (LTS)](./lts/index.md). The two offerings are suited to different sets of use-cases and are available upon request, so please read on to determine which may be most suitable. - -[Project Storage](./cheaha_storage_gpfs/project_directories.md) is best-suited for changing or dynamic data. Specifically:: - -- Data needing/undergoing analysis -- Exploratory data -- Temporary data needed longer than 30 days - -In contrast, [Long-Term Storage](./lts/index.md) is best-suited for unchanging or static data. Specifically: - -- Instrument-acquired data -- Completed analyses -- Hosting data for others to copy -- Hosting data for the public internet -- "Pick-up" and "drop-off" locations for data as part of a workflow - -Shared Storage is available for labs headed by a PI _and_ for Core facilities headed by a director. - -Shared Storage is allocated on a per-organization basis, not on a per-person basis. If an individual researcher manages both a lab and a Core, they may request independent storage allocations for each organization. Each organization may request both Project Storage and Long-Term Storage. - -### Which Platform Do I Need an Allocation For? - -You can request one or both storage platforms, depending on your project’s needs. - -- Request [Cheaha Storage (GPFS)](./cheaha_storage_gpfs/index.md) if you need fast, cluster-integrated storage to support active analysis or collaboration on Cheaha. -- Request [Long-Term Storage (LTS)](./lts/index.md) if you need to store, or share data that does not change frequently. - -You can benefit from requesting and using both platforms: keeping active and changing data in GPFS during analysis, then moving completed or less active data into LTS for long-term preservation, and bringing it back from LTS to GPFS when further work is needed. - -### How Do I Request Shared Storage? - -To request shared Project Storage or Long-Term Storage, please contact [Support](../help/support.md). To ensure prompt allocation of Shared Storage, please follow the guidelines below: - -- Requests must be made to or via the [AskIT HelpDesk](https://www.uab.edu/it/home/). -- Requests must come from one of the proposed owners (a Lab PI, a Research Core director, or both). -- The role of Lab PI entitles a person to a project space for that lab. -- The role of Research Core director entitles a person to a project space for that core. If one person has both roles, they may have two shared Storage spaces, one for each role. -- All proposed owners must have created their [Research Computing accounts](../account_management/cheaha_account.md) at the time of the request. - -Please provide the following information. Missing information can delay allocation of Shared Storage as we either look up the information, or ask followup questions. - -- **Responsible Party/Owner:** The BlazerID of the person claiming responsibility for what happens and what is stored in the space. Typically this would be a Principal Investigator (PI) or a Core Director. - - Multiple responsible parties are allowed. - - We need one person declared as "primary" owner. This person will be the literal owner (in the Linux sense) for Project Storage. -- **Members:** A list of BlazerIDs of people to give access to the space. (Note: this only applies to Project Storage. LTS access controls are managed differently.) -- **Type of Organization:** Is the Shared Storage request for a lab, core, campus administrative group, or something else? -- **Name of Organization:** The _specific_ name of the organization the Shared Storage request is for. -- **Parent Organization:** The name of the parent organization for your organization. Please be as detailed as possible. -- **Purpose of Shared Storage:** The research purpose for the storage, how do you intend to use it? Please feel free to be as detailed as you like, but please limit to a few sentences at most. -- **Internal UAB Collaborator Organizations:** The name(s) of any other UAB organizations participating in the Shared Storage. -- **External Collaborator Organizations:** The name(s) of any external organizations participating in the Shared Storage. -- **Regulatory Requirements:** List any regulatory requirements or agencies affecting data to be stored in the space. Possibilities include, but are not limited to: IRB, EHR, HIPAA, PHI, FERPA. -- **Name of Shared Storage:** Please give us a generic name specific to your project/Lab. - - - For Labs, we recommend using the format `_lab`, where `` is the BlazerID of the Principal Investigator (PI). Alternatively, the PI may choose to use their first or last name instead of the BlazerID. - - For Cores, we recommend using a shortened version of the Core name. For example: `core_facility_space` - - For Project Storage, the name you choose will be used in the path `/data/project/_lab` on Cheaha. Also, this name,`_lab`, will be given to your shared LTS allocation. - - - !!! Tip - - - Keep the name short, memorable, and relevant. - - Use `underscores (_)` or `hyphen (-)` to separate words. - - To serve future projects, consider names that are generic. - - -If some members have not created their accounts at the time of the request, we will proceed with allocating the Shared Storage. Additional members may be added at a later time in a new service request. - -### How Do I Make Changes to Shared Storage Membership? - -To request changes in Shared Storage membership, please contact [Support](../help/support.md). Please take note of the following guidelines to ensure changes can be made promptly. - -- We must have written approval from an owner to make membership changes. -- The exact name of the Shared Storage. If it is Project Storage, the path to the storage location, i.e., `/data/project/...`. -- Please give BlazerIDs of members to add or remove. - -### How Can I Get a Larger `/data/project/` (GPFS) Allocation? - -At this time, due to constraints on total GPFS storage, we are not able to increase `/data/project/` allocations. Please consider batching your analyses by leveraging a combination of [LTS](./lts/index.md) to store raw and/or input data, and [Network Scratch](./cheaha_storage_gpfs/network_scratch.md) for temporary storage of up to 100 TB of data for use during analysis. - -If you wish to have further discussion of options for expanding your GPFS allocation and other workarounds tailored to your workflow, please [Contact Support](../help/support.md). Please also note that project storage is not just for a single project only, it is meant as a storage for multiple projects. - -### How Can I Get a Larger LTS Lab Allocation? - -At this time, due to constraints on total [LTS](./lts/index.md) storage, increasing an LTS allocation requires purchasing additional hardware. Below are some facts about purchasing additional storage nodes. - -- Allocation increases occur by purchasing whole storage nodes. -- Each node has 133 TB of usable storage. -- Nodes are purchased with researcher funds at vendor cost. -- No markups are added to the cost of nodes. -- Purchased nodes are racked with existing hardware in our data centers. -- Purchased nodes are maintained by Research Computing with the same level of service as other hardware. -- Purchased nodes are supported for 5 years from date of purchase, the industry standard for commercial datacenter hardware. -- Once an order is placed with the vendor, we can provide additional storage immediately _if_ free storage is available, regardless of lead-time. - -If you have additional questions _or_ wish to discuss further, please [Contact Support](../help/support.md). - -### If I Can't Get a Larger Allocation, What Alternatives Are There? - -One alternative we recommend is breaking your dataset into batches. A generic, template workflow might be something like below. - -- Copy a batch of data from LTS, or an internet source, to [Network Scratch](./cheaha_storage_gpfs/network_scratch.md). -- Perform analyses on copied data in network scratch. -- Store intermediate or final results in `/data/project/` or LTS. -- Delete copied data from network scratch. -- Start again with the next batch. - -When all batches have been processed, begin processing or aggregating the resulting data. - -If you wish to discuss other alternatives tailored to your workflow, please [Contact Support](../help/support.md). - -### How Can I Effectively Manage My Data? - -Keep files well-organized and clearly named, back up regularly, and archive or delete unused data. - -### How Is My Data Protected Against Disk Failure? - -Please see the "Redundancy" row in the [storage table](#what-type-of-storage-do-i-need) to learn how different storage platforms protect data against disk failure. - -### What Are My Responsibilities for Data Management? - -Periodically review permissions, clean up unused data, and follow institutional Storage and security policies. Please review our [Research Data Responsibilities](./research_data_responsibilities.md#research-data-responsibilities) page for details. - -## Data Responsibilities and Procedures - -### Archival - - -!!! important - - Archival of data is the responsibility of researchers using Cheaha. - - -At this time, Research Computing does not offer a method of archival. If you have need for archival, please feel free to contact [Support](../help/support.md) to start a conversation. - -A possible external resource for archival is available through University of Oklahoma (OU) Supercomputing Center for Education and Research (OSCER). Please see the following link for details: . - -### Backups - - -!!! important - - Backups of data are the responsibility of researchers using Cheaha. - - -A good practice for backing up data is to use the 3-2-1 rule, as [recommended by US-CERT](https://www.cisa.gov/sites/default/files/publications/data_backup_options.pdf): - -- **3**: Keep **3** copies of important data. 1 primary copy for use, 2 backup copies. -- **2**: Store backup copies on **2** different media types to protect from media-specific hazards. -- **1**: Store **1** backup copy offsite, located geographically distant from the primary copy. - -What hazards can cause data loss? - -- Accidental file deletion. - - Example: mistakenly deleting the wrong files when using the [shell command](../workflow_solutions/shell.md#delete-files-and-directories-rm-rmdir) `rm`. - - Files deleted with `rm` or any similar command can not be recovered by us under any circumstances. -- Natural disasters. - - Examples: tornado; hurricane. - - All of our data sits in one geographical location at the UAB Technology Innovation Center (TIC). - - Plans to add geographical data redundancy are being considered. -- Unusable backups. - - Examples: backup software bug; media destroyed; natural disaster at offsite location. - -If you need backup services, please [Contact Us](../help/support.md#how-to-request-support), and we can discuss options based on your specific use case. - -### HIPAA Compliance - -Cheaha is HIPAA compliant and can accept Protected Health Information (PHI) data. Currently, [long-term storage](lts/index.md) is NOT HIPAA compliant but will be in the future. - -For UAB policies surrounding PHI data, please see the following URLs. - -- [Data Classification](https://www.uab.edu/it/home/policies/data-classification/classification-overview) -- [Data Protection and Security Policy](https://secure4.compliancebridge.com/uab/portal/getdoc.php?file=302) -- [Data Access Policy](https://secure4.compliancebridge.com/uab/portal/getdoc.php?file=301) -- [HIPAA Data Policy](https://www.uab.edu/it/home/policies/compliance/hipaa) - - -!!! important - - It is the responsibility of researchers to make sure PHI is accessible _only_ to people on the relevant IRB, with a demonstrated need to know. If PHI is stored in a project directory where some researchers are not on the IRB, their access to those files should be restricted using Access Control Lists (ACLs). Access control should be planned in advance of moving PHI data to Cheaha. If you need assistance setting up ACLs properly, please contact [Support](../help/support.md). - - -Managing PHI data can be challenging. There are experts on Campus who can provide assistance. Please contact [Support](../help/support.md) if you intend to use Research Computing services in combination with PHI and PHI-derived data. +- [Storage Service](./storage/index.md): This section explains the types of storage available, their use cases, and how to access them. We provide multiple storage options for research projects, including: + - [Cheaha Storage (GPFS)](./storage/cheaha_storage_gpfs/index.md). + - [UAB LTS](./storage/lts/index.md). + - [Code Storage](./storage/code_storage.md). + - [Alternative Storage Options](./storage/alternate_storage.md). +- [Data Transfer Tools](./transfer/index.md): Various tools are available for transferring data to, from, and between our storage systems, including: [Globus](./transfer/globus/index.md), [Rclone](./transfer/rclone.md), and [FileZilla](./transfer/filezilla.md). Step-by-step tutorials for [Globus](./transfer/globus/index.md) are also provided to help you transfer data between our storage systems. +- [Research Data Responsibilities](./research_data_responsibilities.md): This section explains the roles and responsibilities in research data management. It covers the responsibilities of researchers and research computing teams, provides guidelines for managing, sharing, and preserving research data, and describes good practices for handling shared allocations. diff --git a/docs/data_management/research_data_responsibilities.md b/docs/data_management/research_data_responsibilities.md index 7ddf9596f..477a1a74c 100644 --- a/docs/data_management/research_data_responsibilities.md +++ b/docs/data_management/research_data_responsibilities.md @@ -10,17 +10,17 @@ A shared allocation is owned by a PI of a Lab or director of Core facility. It i Shared storage owners, staff and students are responsible for overseeing and managing the allocations, including granting access to specific folders. However, Research Computing may provide support in certain cases. For example, if a folder becomes "locked" (i.e., no group members can change its permissions or access it), the owner of the allocation or the folder should submit a request for us to fix the issue. In addition, if you need assistance configuring or reconfiguring permissions, we can provide support as a convenience. Simply send us a request via . -{{ read_csv('data_management/res/cheaha_project_directory.csv', keep_default_na=False) }} +{{ read_csv('data_management/storage/res/cheaha_project_directory.csv', keep_default_na=False) }} ## Individual Allocation Individual allocations are intended for personal or individual use and are available to all UAB affiliated individuals or UAB employee's sponsored Collaborator. It is tied to the individual’s email and provide 5 TB of home/user directory on Cheaha and additional 5 TB of LTS allocation. -{{ read_csv('data_management/res/cheaha_individual_account.csv', keep_default_na=False) }} +{{ read_csv('data_management/storage/res/cheaha_individual_account.csv', keep_default_na=False) }} ## Data Archival and Backup Procedures -Researchers and users of Cheaha are responsible to organize data, archive inactive files, and back up critical data. For backup and archival solutions, please review our [Data Responsibilities and Procedures](./index.md#data-responsibilities-and-procedures) page. If you need backup and Archival assistance, we can discuss options based on your use cases. Please send us a support ticket via . +Researchers and users of Cheaha are responsible to organize data, archive inactive files, and back up critical data. For backup and archival solutions, please review our [Data Responsibilities and Procedures](../data_management/storage/index.md#data-responsibilities-and-procedures) page. If you need backup and Archival assistance, we can discuss options based on your use cases. Please send us a support ticket via . ## Security Exceptions for Accessing Former UAB Personnel Data diff --git a/docs/data_management/alternate_storage.md b/docs/data_management/storage/alternate_storage.md similarity index 100% rename from docs/data_management/alternate_storage.md rename to docs/data_management/storage/alternate_storage.md diff --git a/docs/data_management/cheaha_storage_gpfs/index.md b/docs/data_management/storage/cheaha_storage_gpfs/index.md similarity index 100% rename from docs/data_management/cheaha_storage_gpfs/index.md rename to docs/data_management/storage/cheaha_storage_gpfs/index.md diff --git a/docs/data_management/cheaha_storage_gpfs/individual_directories.md b/docs/data_management/storage/cheaha_storage_gpfs/individual_directories.md similarity index 87% rename from docs/data_management/cheaha_storage_gpfs/individual_directories.md rename to docs/data_management/storage/cheaha_storage_gpfs/individual_directories.md index 27db43fd5..10c96a7ce 100644 --- a/docs/data_management/cheaha_storage_gpfs/individual_directories.md +++ b/docs/data_management/storage/cheaha_storage_gpfs/individual_directories.md @@ -32,7 +32,7 @@ Quota reports are updated nightly, so they may be out of date if you move data a This section outlines key storage policies and exceptions, including access control, data management, and conditions for administrative access or quota increases in user data and home directories. - Storage Entitlement: All users receive a 5TB of default quota upon creating a Cheaha account. -- Quota Increase: At this time, we do not provide a quota increase in user data and home directories. If you require more storage for your work, please consider using [shared project allocation](./project_directories.md) or an [LTS](../../data_management/lts/index.md). +- Quota Increase: At this time, we do not provide a quota increase in user data and home directories. If you require more storage for your work, please consider using [shared project allocation](./project_directories.md) or an [LTS](../lts/index.md). - Data Management: Users are responsible for organizing, cleaning, and backing up their data in their user data and home directory. -- Permission and Access Control Management: Access in user data and home directories can be managed using Access Control Lists (ACLs). Where third-party access is required, a security exception may apply. Please refer to the [Security Exception](../../data_management/research_data_responsibilities.md#security-exceptions-for-accessing-former-uab-personnel-data) page for more details. -- HIPAA Compliance: Users data and home directories are HIPAA compliant and can accept Protected Health Information (PHI) data. For HIPAA Compliance and UAB policies surrounding PHI data, please refer the [HIPAA Compliance](../../data_management/index.md#hipaa-compliance)page. +- Permission and Access Control Management: Access in user data and home directories can be managed using Access Control Lists (ACLs). Where third-party access is required, a security exception may apply. Please refer to the [Security Exception](../../research_data_responsibilities.md#security-exceptions-for-accessing-former-uab-personnel-data) page for more details. +- HIPAA Compliance: Users data and home directories are HIPAA compliant and can accept Protected Health Information (PHI) data. For HIPAA Compliance and UAB policies surrounding PHI data, please refer the [HIPAA Compliance](../../storage/index.md#hipaa-compliance)page. diff --git a/docs/data_management/cheaha_storage_gpfs/local_scratch.md b/docs/data_management/storage/cheaha_storage_gpfs/local_scratch.md similarity index 87% rename from docs/data_management/cheaha_storage_gpfs/local_scratch.md rename to docs/data_management/storage/cheaha_storage_gpfs/local_scratch.md index d2a5f075d..20624660c 100644 --- a/docs/data_management/cheaha_storage_gpfs/local_scratch.md +++ b/docs/data_management/storage/cheaha_storage_gpfs/local_scratch.md @@ -8,11 +8,11 @@ At this time you will need to make local scratch subdirectories yourself with `m Some known examples of tasks benefiting from local scratch, not an exhaustive list: -- AI and deep learning training on [A100 GPUs](../../cheaha/slurm/gpu.md). +- AI and deep learning training on [A100 GPUs](../../../cheaha/slurm/gpu.md). - Large-scale genome annotation. - Reading/writing hundreds of thousands or more files in a single job. -If you are using `amperenodes` and the A100 GPUs, then you should use local scratch for your data to ensure adequate GPU performance. Using [network scratch](./network_scratch.md), or other network file locations, will starve the GPU of data, resulting in poor GPU performance. For more information please see [Ensuring IO Performance With A100 GPUs](../../cheaha/slurm/gpu.md#ensuring-io-performance-with-a100-gpus). +If you are using `amperenodes` and the A100 GPUs, then you should use local scratch for your data to ensure adequate GPU performance. Using [network scratch](./network_scratch.md), or other network file locations, will starve the GPU of data, resulting in poor GPU performance. For more information please see [Ensuring IO Performance With A100 GPUs](../../../cheaha/slurm/gpu.md#ensuring-io-performance-with-a100-gpus). !!! important @@ -20,7 +20,7 @@ If you are using `amperenodes` and the A100 GPUs, then you should use local scra Be sure to clean up `/local/$USER/$SLURM_JOB_ID` after your job is complete! -An example script to automate this process is shown below. This example shows how you can wrap your workflow with deployment and cleanup of local scratch. The following sample script only applies if you are running a small number of jobs (less than one hundred). If you need to run many jobs all using the same data, such as with a large array using the `--array` flag, please [contact us](../../help/support.md) about preloading the data onto your desired nodes. This will avoid the per-job overhead of copying and deleting files. +An example script to automate this process is shown below. This example shows how you can wrap your workflow with deployment and cleanup of local scratch. The following sample script only applies if you are running a small number of jobs (less than one hundred). If you need to run many jobs all using the same data, such as with a large array using the `--array` flag, please [contact us](../../../help/support.md) about preloading the data onto your desired nodes. This will avoid the per-job overhead of copying and deleting files. ```bash #!/bin/bash @@ -52,7 +52,7 @@ rm -rf "$TMPDIR" !!! important - Using `/local/$USER/$SLURM_JOB_ID` with MPI jobs takes additional consideration. If you do not need MPI, please use the `#SBATCH --nodes=1` slurm directive to specify that all requested cores are on the same node. If you need the performance of `/local/$USER/$SLURM_JOB_ID` in an MPI job, please contact [Support](../../help/support.md) and read about the Slurm commands `sbcast` and `sgather`. + Using `/local/$USER/$SLURM_JOB_ID` with MPI jobs takes additional consideration. If you do not need MPI, please use the `#SBATCH --nodes=1` slurm directive to specify that all requested cores are on the same node. If you need the performance of `/local/$USER/$SLURM_JOB_ID` in an MPI job, please contact [Support](../../../help/support.md) and read about the Slurm commands `sbcast` and `sgather`. @@ -65,8 +65,8 @@ rm -rf "$TMPDIR" ## What if My Data Won't Fit in Local Scratch? -Be sure that your files will fit in `/local/` before starting. You can determine disk size and current usage using `df -h | grep "local"`. Most nodes have 1.0 TB total capacity, while the `amperenodes` have 6.0 TB. If you data won't fit in the current usage, or on the drives, please [Contact Us](../../help/support.md). We can work with you to identify a solution. +Be sure that your files will fit in `/local/` before starting. You can determine disk size and current usage using `df -h | grep "local"`. Most nodes have 1.0 TB total capacity, while the `amperenodes` have 6.0 TB. If you data won't fit in the current usage, or on the drives, please [Contact Us](../../../help/support.md). We can work with you to identify a solution. ## What if I Have a Large Amount of Data for Local Scratch? -If you have a large amount of data but each job takes very little time to run, performance can be further improved by avoiding frequent data copies and deletions. In these cases, preloading the data onto local scratch only once and then reusing it makes more sense. If this is the case for you, or you think you might benefit, please [Contact Us](../../help/support.md) and we can discuss creating a temporary node reservation to allow one-time data preloading. +If you have a large amount of data but each job takes very little time to run, performance can be further improved by avoiding frequent data copies and deletions. In these cases, preloading the data onto local scratch only once and then reusing it makes more sense. If this is the case for you, or you think you might benefit, please [Contact Us](../../../help/support.md) and we can discuss creating a temporary node reservation to allow one-time data preloading. diff --git a/docs/data_management/cheaha_storage_gpfs/network_scratch.md b/docs/data_management/storage/cheaha_storage_gpfs/network_scratch.md similarity index 100% rename from docs/data_management/cheaha_storage_gpfs/network_scratch.md rename to docs/data_management/storage/cheaha_storage_gpfs/network_scratch.md diff --git a/docs/data_management/cheaha_storage_gpfs/project_directories.md b/docs/data_management/storage/cheaha_storage_gpfs/project_directories.md similarity index 89% rename from docs/data_management/cheaha_storage_gpfs/project_directories.md rename to docs/data_management/storage/cheaha_storage_gpfs/project_directories.md index 87433aa36..a4206275f 100644 --- a/docs/data_management/cheaha_storage_gpfs/project_directories.md +++ b/docs/data_management/storage/cheaha_storage_gpfs/project_directories.md @@ -14,11 +14,11 @@ Every project directory has a group that is unique system-wide, and not used any Some early group names may not match their project directory, but should be reasonably close. -Members of the project directory group have permissions to access that project directory. Adding and removing members from the project directory group is how Research Computing controls access to, and ownership of, project directories. We do not use access control lists (ACLs) to manage permissions ourselves, but use of ACLs is allowed and encouraged for PIs and project administrators who want more fine-grained control. Please see our [section on ACLs](../../workflow_solutions/shell.md#manage-researcher-access-to-files-and-directories-getfacl-setfacl) for more information. +Members of the project directory group have permissions to access that project directory. Adding and removing members from the project directory group is how Research Computing controls access to, and ownership of, project directories. We do not use access control lists (ACLs) to manage permissions ourselves, but use of ACLs is allowed and encouraged for PIs and project administrators who want more fine-grained control. Please see our [section on ACLs](../../../workflow_solutions/shell.md#manage-researcher-access-to-files-and-directories-getfacl-setfacl) for more information. Be default, project space permissions are set up in the following way: -{{ read_csv('data_management/res/project_space_permissions.csv', keep_default_na=False) }} +{{ read_csv('data_management/storage/res/project_space_permissions.csv', keep_default_na=False) }} Having `setgid` enabled on directories means new files and directories created within will inherit group ownership and the `setgid` bit. The `setgid` bit is reflected by the `2` in the numeric permissions and the `s` in the symbolic permissions. The `setgid` bit and per-directory project groups is how Research Computing controls access to each project directory. @@ -35,7 +35,7 @@ There are some known issues surrounding project directory permissions when files For PIs and project administrators: - Please educate your staff and collaborators about the above permission setups, and any additional ACLs you may have in place, to minimize future challenges. -- If you have issues with permissions, please contact [Support](../../help/support.md). We can guide you through [Managing Permissions](../../workflow_solutions/shell.md#manage-permissions-of-files-and-directores-chmod) and [Managing Group Ownership](../../workflow_solutions/shell.md#manage-group-ownership-chgrp). +- If you have issues with permissions, please contact [Support](../../../help/support.md). We can guide you through [Managing Permissions](../../../workflow_solutions/shell.md#manage-permissions-of-files-and-directores-chmod) and [Managing Group Ownership](../../../workflow_solutions/shell.md#manage-group-ownership-chgrp). ## Project Directory Organization @@ -118,9 +118,9 @@ To provide a clearer picture, here is how all the components fit together in the ## Permissions and Access Control -Permissions for each directory can be managed by either being a member of the group associated with the top-level project directory or by using [Linux permissions](../cheaha_storage_gpfs/project_directories.md#project-directory-permissions) or [Access Control Lists (ACLs)](../../workflow_solutions/shell.md#manage-researcher-access-to-files-and-directories-getfacl-setfacl). Managing proper permissions ensures that team members can access only the resources they need, while protecting sensitive data. +Permissions for each directory can be managed by either being a member of the group associated with the top-level project directory or by using [Linux permissions](../cheaha_storage_gpfs/project_directories.md#project-directory-permissions) or [Access Control Lists (ACLs)](../../../workflow_solutions/shell.md#manage-researcher-access-to-files-and-directories-getfacl-setfacl). Managing proper permissions ensures that team members can access only the resources they need, while protecting sensitive data. -If you plan managing multiple projects and would like assistance with organizing your project directories or managing permissions, please send us a [support ticket](../../help/support.md#how-do-i-create-a-support-ticket). +If you plan managing multiple projects and would like assistance with organizing your project directories or managing permissions, please send us a [support ticket](../../../help/support.md#how-do-i-create-a-support-ticket). ## How Much Space Do I Have Left? @@ -131,7 +131,7 @@ To check the storage space left from your project directory, use the command `pr This section outlines key storage policies and exceptions, including access control and data management, conditions for administrative access or quota increases in Project directory. - Storage Entitlement: A shared project allocation of 25 TB is available to research labs led by a PI and to Core Facilities headed by a Director. A shared project allocation is intended to support active research and collaboration. -- Quota Increase: At this time, we do not provide additional quota increase for shared project allocation. If you require more storage for your work, please consider requesting and using an [LTS](../../data_management/lts/index.md). +- Quota Increase: At this time, we do not provide additional quota increase for shared project allocation. If you require more storage for your work, please consider requesting and using an [LTS](../lts/index.md). - Data Management: The owner the shared Project allocation is responsible for organizing, cleaning, and backing up their data in their shared Project allocation. - Permission and Access Control Management: Access to shared project allocation can be managed using Access Control Lists (ACLs). New members can be added to or removed from the group access based on PI approval. -- HIPAA Compliance: Shared project allocation is HIPAA compliant and can accept Protected Health Information (PHI) data. For HIPAA Compliance and UAB policies surrounding PHI data, please refer the [HIPAA Compliance](../../data_management/index.md#hipaa-compliance)page. +- HIPAA Compliance: Shared project allocation is HIPAA compliant and can accept Protected Health Information (PHI) data. For HIPAA Compliance and UAB policies surrounding PHI data, please refer the [HIPAA Compliance](../index.md#hipaa-compliance)page. diff --git a/docs/data_management/cheaha_storage_gpfs/temporary_files.md b/docs/data_management/storage/cheaha_storage_gpfs/temporary_files.md similarity index 100% rename from docs/data_management/cheaha_storage_gpfs/temporary_files.md rename to docs/data_management/storage/cheaha_storage_gpfs/temporary_files.md diff --git a/docs/data_management/code_storage.md b/docs/data_management/storage/code_storage.md similarity index 85% rename from docs/data_management/code_storage.md rename to docs/data_management/storage/code_storage.md index f839f5f13..d81f5d7f6 100644 --- a/docs/data_management/code_storage.md +++ b/docs/data_management/storage/code_storage.md @@ -8,14 +8,14 @@ If your work is private, or internal to UAB operations, it is probably best to s ### GitLab -UAB Research Computing maintains an on-premises GitLab instance. To gain access, please visit and create an [Account](../account_management/gitlab_account.md). +UAB Research Computing maintains an on-premises GitLab instance. To gain access, please visit and create an [Account](../../account_management/gitlab_account.md). ### GitHub Please visit . -For UAB Research Computing's GitHub repositories, please see our [Social Media](../education/training_resources.md#our-social-media-presence) page. +For UAB Research Computing's GitHub repositories, please see our [Social Media](../../education/training_resources.md#our-social-media-presence) page. ## How Do I Effectively Use Git, GitHub, and GitLab for Collaboration? -Please see our page on [Collaborating with Git](../workflow_solutions/git_collaboration.md). +Please see our page on [Collaborating with Git](../../workflow_solutions/git_collaboration.md). diff --git a/docs/data_management/images/box_acc_icon.png b/docs/data_management/storage/images/box_acc_icon.png similarity index 100% rename from docs/data_management/images/box_acc_icon.png rename to docs/data_management/storage/images/box_acc_icon.png diff --git a/docs/data_management/images/box_stor_size.png b/docs/data_management/storage/images/box_stor_size.png similarity index 100% rename from docs/data_management/images/box_stor_size.png rename to docs/data_management/storage/images/box_stor_size.png diff --git a/docs/data_management/storage/index.md b/docs/data_management/storage/index.md new file mode 100644 index 000000000..73ee9e1fb --- /dev/null +++ b/docs/data_management/storage/index.md @@ -0,0 +1,226 @@ +# Storage + +Research Computing offers several data storage options to meet individual or shared needs of UAB researchers, depending on their requirement and use-cases. The types of storage available, procedures for requesting access, responsibilities, and usage guidelines are detailed in the following sections. + +## What Type of Storage Do I Need? + +There are multiple locations for data storage both on and off Cheaha each with a specific purpose. You can look at the table below to help determine the storage platform we provide that best matches your needed use-case. After reviewing the table, proceed to the [How to Choose the Right Storage for My Use Case](#how-to-choose-the-right-storage-for-my-use-case) section for additional guidance on selecting the most suitable storage for your needs. If you need additional assistance, please contact [Support](../../help/support.md). + +{{ read_csv('data_management/storage/res/storage_overview.csv', keep_default_na=False) }} + +### How to Choose the Right Storage for My Use Case? + +When you have different types of data with varying storage needs, choosing the right storage is not an easy task. Below, we will walk you through common data storage needs and use cases. + +- **I have lots of data that rarely or never changes**: + - If your data is large and does not change frequently, [Long-Term Storage (LTS)](./lts/index.md) is your ideal option. +- **I need to do a lot of processing on the data**: + - If your data is frequently accessed or modified, [Cheaha Storage (GPFS)](./cheaha_storage_gpfs/index.md) is a perfect option for your use case. + - When you have finished working with the data, please consider putting it into [LTS](./lts/index.md). Please [Contact Us](../../help/support.md#how-to-request-support) for more information. +- **I am just learning how to use Cheaha for very small personal projects**: + - If you are new to Cheaha and working with small personal projects, you do not need a complex storage solution. [User Data](./cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories) is a simple and accessible option for beginners and smaller datasets. + - If you find your project growing, or you need to collaborate with others at UAB, consider a shared allocation on [GPFS](./cheaha_storage_gpfs/index.md) and/or [LTS](./lts/index.md). Please refer to the instructions on [how to request a shared allocation](#how-do-i-request-shared-storage). +- **I need somewhere to store lots of temporary files while I do my processing**: + - If you need a place to store large amounts of temporary data while processing data, [Network Scratch](./cheaha_storage_gpfs/network_scratch.md) is the best option. +- **I am working with high-throughput or I/O bound applications, like AI training or evaluation, or I have many very small files**: + - For high-performance tasks, like AI training, [Local Scratch](./cheaha_storage_gpfs/local_scratch.md) provides fast and optimized data access for intensive I/O workloads.. +- **I have a combination of needs and am unsure of how to proceed**: + - If your use case spans across multiple types of storage needs, [Contact Us](../../help/support.md#how-to-request-support) to discuss your requirements, and we wll help you find the best solution. + +## What Individual Storage Solutions Are Available? + +Every Cheaha user has personal directories found at `/home/$USER` (or `$HOME`) and `/data/user/$USER` (or `$USER_DATA`), which are created automatically during account registration. In addition, individual allocations on Long-Term Storage (LTS) are also available upon request. Please read more about [Long-Term Storage](./lts/index.md) and [User Data and Home Directories](./cheaha_storage_gpfs/individual_directories.md#home-and-user-data-directories). + +### How Do I Request Individual Cheaha (`GPFS`) Storage? + +An individual Cheaha (`GPFS`) storage is automatically created as part of the Cheaha account setup. No separate request is required. + +### How Do I Request Individual Long-Term Storage? + +To request individual Long-Term Storage, please first read and understand how [Long-Term Storage](./lts/index.md) differs from traditional file systems, like GPFS on Cheaha. Decide if it is suitable for your needs. Then please feel free to contact [Support](../../help/support.md). + +## What Shared Storage Solutions Are Available? + +Shared Storage is available via two services. We have [Project Storage](./cheaha_storage_gpfs/project_directories.md) (located in `/data/project` or Cheaha) and [Long-Term Storage (LTS)](./lts/index.md). The two offerings are suited to different sets of use-cases and are available upon request, so please read on to determine which may be most suitable. + +[Project Storage](./cheaha_storage_gpfs/project_directories.md) is best-suited for changing or dynamic data. Specifically:: + +- Data needing/undergoing analysis +- Exploratory data +- Temporary data needed longer than 30 days + +In contrast, [Long-Term Storage](./lts/index.md) is best-suited for unchanging or static data. Specifically: + +- Instrument-acquired data +- Completed analyses +- Hosting data for others to copy +- Hosting data for the public internet +- "Pick-up" and "drop-off" locations for data as part of a workflow + +Shared Storage is available for labs headed by a PI _and_ for Core facilities headed by a director. + +Shared Storage is allocated on a per-organization basis, not on a per-person basis. If an individual researcher manages both a lab and a Core, they may request independent storage allocations for each organization. Each organization may request both Project Storage and Long-Term Storage. + +### Which Platform Do I Need an Allocation For? + +You can request one or both storage platforms, depending on your project’s needs. + +- Request [Cheaha Storage (GPFS)](./cheaha_storage_gpfs/index.md) if you need fast, cluster-integrated storage to support active analysis or collaboration on Cheaha. +- Request [Long-Term Storage (LTS)](./lts/index.md) if you need to store, or share data that does not change frequently. + +You can benefit from requesting and using both platforms: keeping active and changing data in GPFS during analysis, then moving completed or less active data into LTS for long-term preservation, and bringing it back from LTS to GPFS when further work is needed. + +### How Do I Request Shared Storage? + +To request shared Project Storage or Long-Term Storage, please contact [Support](../../help/support.md). To ensure prompt allocation of Shared Storage, please follow the guidelines below: + +- Requests must be made to or via the [AskIT HelpDesk](https://www.uab.edu/it/home/). +- Requests must come from one of the proposed owners (a Lab PI, a Research Core director, or both). +- The role of Lab PI entitles a person to a project space for that lab. +- The role of Research Core director entitles a person to a project space for that core. If one person has both roles, they may have two shared Storage spaces, one for each role. +- All proposed owners must have created their [Research Computing accounts](../../account_management/cheaha_account.md) at the time of the request. + +Please provide the following information. Missing information can delay allocation of Shared Storage as we either look up the information, or ask followup questions. + +- **Responsible Party/Owner:** The BlazerID of the person claiming responsibility for what happens and what is stored in the space. Typically this would be a Principal Investigator (PI) or a Core Director. + - Multiple responsible parties are allowed. + - We need one person declared as "primary" owner. This person will be the literal owner (in the Linux sense) for Project Storage. +- **Members:** A list of BlazerIDs of people to give access to the space. (Note: this only applies to Project Storage. LTS access controls are managed differently.) +- **Type of Organization:** Is the Shared Storage request for a lab, core, campus administrative group, or something else? +- **Name of Organization:** The _specific_ name of the organization the Shared Storage request is for. +- **Parent Organization:** The name of the parent organization for your organization. Please be as detailed as possible. +- **Purpose of Shared Storage:** The research purpose for the storage, how do you intend to use it? Please feel free to be as detailed as you like, but please limit to a few sentences at most. +- **Internal UAB Collaborator Organizations:** The name(s) of any other UAB organizations participating in the Shared Storage. +- **External Collaborator Organizations:** The name(s) of any external organizations participating in the Shared Storage. +- **Regulatory Requirements:** List any regulatory requirements or agencies affecting data to be stored in the space. Possibilities include, but are not limited to: IRB, EHR, HIPAA, PHI, FERPA. +- **Name of Shared Storage:** Please give us a generic name specific to your project/Lab. + + - For Labs, we recommend using the format `_lab`, where `` is the BlazerID of the Principal Investigator (PI). Alternatively, the PI may choose to use their first or last name instead of the BlazerID. + - For Cores, we recommend using a shortened version of the Core name. For example: `core_facility_space` + - For Project Storage, the name you choose will be used in the path `/data/project/_lab` on Cheaha. Also, this name,`_lab`, will be given to your shared LTS allocation. + + + !!! Tip + + - Keep the name short, memorable, and relevant. + - Use `underscores (_)` or `hyphen (-)` to separate words. + - To serve future projects, consider names that are generic. + + +If some members have not created their accounts at the time of the request, we will proceed with allocating the Shared Storage. Additional members may be added at a later time in a new service request. + +### How Do I Make Changes to Shared Storage Membership? + +To request changes in Shared Storage membership, please contact [Support](../../help/support.md). Please take note of the following guidelines to ensure changes can be made promptly. + +- We must have written approval from an owner to make membership changes. +- The exact name of the Shared Storage. If it is Project Storage, the path to the storage location, i.e., `/data/project/...`. +- Please give BlazerIDs of members to add or remove. + +### How Can I Get a Larger `/data/project/` (GPFS) Allocation? + +At this time, due to constraints on total GPFS storage, we are not able to increase `/data/project/` allocations. Please consider batching your analyses by leveraging a combination of [LTS](./lts/index.md) to store raw and/or input data, and [Network Scratch](./cheaha_storage_gpfs/network_scratch.md) for temporary storage of up to 100 TB of data for use during analysis. + +If you wish to have further discussion of options for expanding your GPFS allocation and other workarounds tailored to your workflow, please [Contact Support](../../help/support.md). Please also note that project storage is not just for a single project only, it is meant as a storage for multiple projects. + +### How Can I Get a Larger LTS Lab Allocation? + +At this time, due to constraints on total [LTS](./lts/index.md) storage, increasing an LTS allocation requires purchasing additional hardware. Below are some facts about purchasing additional storage nodes. + +- Allocation increases occur by purchasing whole storage nodes. +- Each node has 133 TB of usable storage. +- Nodes are purchased with researcher funds at vendor cost. +- No markups are added to the cost of nodes. +- Purchased nodes are racked with existing hardware in our data centers. +- Purchased nodes are maintained by Research Computing with the same level of service as other hardware. +- Purchased nodes are supported for 5 years from date of purchase, the industry standard for commercial datacenter hardware. +- Once an order is placed with the vendor, we can provide additional storage immediately _if_ free storage is available, regardless of lead-time. + +If you have additional questions _or_ wish to discuss further, please [Contact Support](../../help/support.md). + +### If I Can't Get a Larger Allocation, What Alternatives Are There? + +One alternative we recommend is breaking your dataset into batches. A generic, template workflow might be something like below. + +- Copy a batch of data from LTS, or an internet source, to [Network Scratch](./cheaha_storage_gpfs/network_scratch.md). +- Perform analyses on copied data in network scratch. +- Store intermediate or final results in `/data/project/` or LTS. +- Delete copied data from network scratch. +- Start again with the next batch. + +When all batches have been processed, begin processing or aggregating the resulting data. + +If you wish to discuss other alternatives tailored to your workflow, please [Contact Support](../../help/support.md). + +### How Can I Effectively Manage My Data? + +Keep files well-organized and clearly named, back up regularly, and archive or delete unused data. + +### How Is My Data Protected Against Disk Failure? + +Please see the "Redundancy" row in the [storage table](#what-type-of-storage-do-i-need) to learn how different storage platforms protect data against disk failure. + +### What Are My Responsibilities for Data Management? + +Periodically review permissions, clean up unused data, and follow institutional Storage and security policies. Please review our [Research Data Responsibilities](../research_data_responsibilities.md#research-data-responsibilities) page for details. + +## Data Responsibilities and Procedures + +### Archival + + +!!! important + + Archival of data is the responsibility of researchers using Cheaha. + + +At this time, Research Computing does not offer a method of archival. If you have need for archival, please feel free to contact [Support](../../help/support.md) to start a conversation. + +A possible external resource for archival is available through University of Oklahoma (OU) Supercomputing Center for Education and Research (OSCER). Please see the following link for details: . + +### Backups + + +!!! important + + Backups of data are the responsibility of researchers using Cheaha. + + +A good practice for backing up data is to use the 3-2-1 rule, as [recommended by US-CERT](https://www.cisa.gov/sites/default/files/publications/data_backup_options.pdf): + +- **3**: Keep **3** copies of important data. 1 primary copy for use, 2 backup copies. +- **2**: Store backup copies on **2** different media types to protect from media-specific hazards. +- **1**: Store **1** backup copy offsite, located geographically distant from the primary copy. + +What hazards can cause data loss? + +- Accidental file deletion. + - Example: mistakenly deleting the wrong files when using the [shell command](../../workflow_solutions/shell.md#delete-files-and-directories-rm-rmdir) `rm`. + - Files deleted with `rm` or any similar command can not be recovered by us under any circumstances. +- Natural disasters. + - Examples: tornado; hurricane. + - All of our data sits in one geographical location at the UAB Technology Innovation Center (TIC). + - Plans to add geographical data redundancy are being considered. +- Unusable backups. + - Examples: backup software bug; media destroyed; natural disaster at offsite location. + +If you need backup services, please [Contact Us](../../help/support.md#how-to-request-support), and we can discuss options based on your specific use case. + +### HIPAA Compliance + +Cheaha is HIPAA compliant and can accept Protected Health Information (PHI) data. Currently, [long-term storage](lts/index.md) is NOT HIPAA compliant but will be in the future. + +For UAB policies surrounding PHI data, please see the following URLs. + +- [Data Classification](https://www.uab.edu/it/home/policies/data-classification/classification-overview) +- [Data Protection and Security Policy](https://secure4.compliancebridge.com/uab/portal/getdoc.php?file=302) +- [Data Access Policy](https://secure4.compliancebridge.com/uab/portal/getdoc.php?file=301) +- [HIPAA Data Policy](https://www.uab.edu/it/home/policies/compliance/hipaa) + + +!!! important + + It is the responsibility of researchers to make sure PHI is accessible _only_ to people on the relevant IRB, with a demonstrated need to know. If PHI is stored in a project directory where some researchers are not on the IRB, their access to those files should be restricted using Access Control Lists (ACLs). Access control should be planned in advance of moving PHI data to Cheaha. If you need assistance setting up ACLs properly, please contact [Support](../../help/support.md). + + +Managing PHI data can be challenging. There are experts on Campus who can provide assistance. Please contact [Support](../../help/support.md) if you intend to use Research Computing services in combination with PHI and PHI-derived data. diff --git a/docs/data_management/lts/iam_and_policies.md b/docs/data_management/storage/lts/iam_and_policies.md similarity index 93% rename from docs/data_management/lts/iam_and_policies.md rename to docs/data_management/storage/lts/iam_and_policies.md index 7dcbf2c26..015febdfc 100644 --- a/docs/data_management/lts/iam_and_policies.md +++ b/docs/data_management/storage/lts/iam_and_policies.md @@ -32,9 +32,9 @@ Everyone is responsible for managing their own keys and ensuring they use the co ### How Can I Recover My Keys? -If you have lost your LTS keys, you can request a reset by creating a support ticket via [Contact Us](../../index.md#how-to-contact-us). Please include your BlazerID and specify the LTS allocation (individual and/or shared) for which you need the key reset, so we can process your request accordingly. Then you will receive an email with a link to a UAB Box text file containing the corresponding keys (access key and secret key). +If you have lost your LTS keys, you can request a reset by creating a support ticket via [Contact Us](../../../index.md#how-to-contact-us). Please include your BlazerID and specify the LTS allocation (individual and/or shared) for which you need the key reset, so we can process your request accordingly. Then you will receive an email with a link to a UAB Box text file containing the corresponding keys (access key and secret key). -If you, as a Lab/Core PI, do not wish to manage the LTS space yourself, we recommend assigning data Steward permissions to someone who is both trustworthy and has knowledge of, or willingness to learn, [JSON](https://docs.fileformat.com/web/json/#google_vignette) and parts of the [Amazon AWS S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_Reference.html). If you need help or have concerns about making this decision, please [Contact Us](../../index.md#how-to-contact-us). +If you, as a Lab/Core PI, do not wish to manage the LTS space yourself, we recommend assigning data Steward permissions to someone who is both trustworthy and has knowledge of, or willingness to learn, [JSON](https://docs.fileformat.com/web/json/#google_vignette) and parts of the [Amazon AWS S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_Reference.html). If you need help or have concerns about making this decision, please [Contact Us](../../../index.md#how-to-contact-us). ## IAM Names @@ -46,21 +46,21 @@ IAM Names are case-sensitive identifiers for allocations and allow granting othe ## Individual LTS Allocations -Individual LTS allocations are intended for personal use and are available to all UAB affiliated individuals and their external collaborators (via a [XIAS account](../../account_management/xias/index.md)). These allocations are tied to the individual’s primary UAB email and provide 5 TB of storage. +Individual LTS allocations are intended for personal use and are available to all UAB affiliated individuals and their external collaborators (via a [XIAS account](../../../account_management/xias/index.md)). These allocations are tied to the individual’s primary UAB email and provide 5 TB of storage. We recommend following the [Individual LTS Tutorial](./tutorial/individual_lts_tutorial.md) to familiarize yourself with working with LTS at the command line. -{{ read_csv('data_management/res/individual_allocation_functional_roles.csv', keep_default_na=False) }} +{{ read_csv('data_management/storage/res/individual_allocation_functional_roles.csv', keep_default_na=False) }} ## Shared LTS Allocations -Shared LTS allocations are data storage designed for collaborative use by groups and are available to Lab PIs and Core Directors. If a PI is also a Core Director, they are eligible for independent storage allocations for each organization: one for the Lab and one for the Core. Each shared LTS allocation provides 75 TB of storage. These allocations can be named according to the preference of the Lab PI or Core Director. For recommended naming guidelines, refer to our [Naming Shared Storage](../../data_management/index.md#how-do-i-request-shared-storage) documentation. +Shared LTS allocations are data storage designed for collaborative use by groups and are available to Lab PIs and Core Directors. If a PI is also a Core Director, they are eligible for independent storage allocations for each organization: one for the Lab and one for the Core. Each shared LTS allocation provides 75 TB of storage. These allocations can be named according to the preference of the Lab PI or Core Director. For recommended naming guidelines, refer to our [Naming Shared Storage](../index.md#how-do-i-request-shared-storage) documentation. -{{ read_csv('data_management/res/shared_allocation_functional_roles.csv', keep_default_na=False) }} +{{ read_csv('data_management/storage/res/shared_allocation_functional_roles.csv', keep_default_na=False) }} ### How Do I Assign a Steward? -Owners can assign stewards either when requesting LTS allocation creation or at a later time by sending a request via [Contact Us](../../index.md#how-to-contact-us). The request should include the steward's BlazerID and specify the LTS allocation they should manage. Once assigned, stewards will have the same management permissions as the owner, except for the ability to assign other stewards. +Owners can assign stewards either when requesting LTS allocation creation or at a later time by sending a request via [Contact Us](../../../index.md#how-to-contact-us). The request should include the steward's BlazerID and specify the LTS allocation they should manage. Once assigned, stewards will have the same management permissions as the owner, except for the ability to assign other stewards. ### Who Can Have What Role? @@ -84,7 +84,7 @@ Owners can assign stewards either when requesting LTS allocation creation or at There are multiple ways to share data with LTS: - With [Bucket Policies](#bucket-policies) -- Using [Globus Guest Collections](../transfer/tutorial/globus_organization_tutorial.md#how-do-i-share-a-collection-with-others) +- Using [Globus Guest Collections](../../transfer/globus/globus_organization_tutorial.md#how-do-i-share-a-collection-with-others) !!! note @@ -432,7 +432,7 @@ s3cmd delpolicy s3:// !!! important - Policies can be very complicated depending on how many people need access to the bucket and how you want to tier permissions (i.e. which people are read-only, read-write, admin-esq priveleges, etc.). If you need help structuring your policy files please [visit us during office hours](../../help/support.md#office-hours) and we will be happy to help structure your policy file to your needs. + Policies can be very complicated depending on how many people need access to the bucket and how you want to tier permissions (i.e. which people are read-only, read-write, admin-esq priveleges, etc.). If you need help structuring your policy files please [visit us during office hours](../../../help/support.md#office-hours) and we will be happy to help structure your policy file to your needs. #### Admin-Like Priveleges @@ -447,4 +447,4 @@ For Labs using LTS to store data from their Cheaha Project Storage directory, it Some groups on campus may distribute datasets to other research groups using LTS. If you are distributing data to multiple groups, and those groups should not have access to each other's data, it is highly advised to store those datasets in separate buckets as opposed to separate directories in a single bucket. -An idiosyncrasy of buckets involves the fact that all objects are stored in the top level of the bucket, and once permissions are given to someone to see the bucket, they will be able to see all objects within the bucket without restrictions even if they are not given download permissions for some objects. If any identifying or priveleged information is given in file names on LTS, it could constitute an IRB violation. Additionally, managing permissions for groups to access data only from specific folders makes the policy file much more complicated and prone to errors. When sharing multiple datasets with multiple different groups, it's advised to keep these data in separate buckets and have individual policy files for each bucket to make policy management simpler and less prone to error. +An idiosyncrasy of buckets involves the fact that all objects are stored in the top level of the bucket, and once permissions are given to someone to see the bucket, they will be able to see all objects within the bucket without restrictions even if they are not given download permissions for some objects. If any identifying or privileged information is given in file names on LTS, it could constitute an IRB violation. Additionally, managing permissions for groups to access data only from specific folders makes the policy file much more complicated and prone to errors. When sharing multiple datasets with multiple different groups, it's advised to keep these data in separate buckets and have individual policy files for each bucket to make policy management simpler and less prone to error. diff --git a/docs/data_management/lts/images/config-s3cmd.png b/docs/data_management/storage/lts/images/config-s3cmd.png similarity index 100% rename from docs/data_management/lts/images/config-s3cmd.png rename to docs/data_management/storage/lts/images/config-s3cmd.png diff --git a/docs/data_management/lts/images/create-bucket.png b/docs/data_management/storage/lts/images/create-bucket.png similarity index 100% rename from docs/data_management/lts/images/create-bucket.png rename to docs/data_management/storage/lts/images/create-bucket.png diff --git a/docs/data_management/lts/images/globus-bucket.png b/docs/data_management/storage/lts/images/globus-bucket.png similarity index 100% rename from docs/data_management/lts/images/globus-bucket.png rename to docs/data_management/storage/lts/images/globus-bucket.png diff --git a/docs/data_management/lts/images/globus-transfer-from-core.png b/docs/data_management/storage/lts/images/globus-transfer-from-core.png similarity index 100% rename from docs/data_management/lts/images/globus-transfer-from-core.png rename to docs/data_management/storage/lts/images/globus-transfer-from-core.png diff --git a/docs/data_management/lts/images/my-lab.png b/docs/data_management/storage/lts/images/my-lab.png similarity index 100% rename from docs/data_management/lts/images/my-lab.png rename to docs/data_management/storage/lts/images/my-lab.png diff --git a/docs/data_management/lts/images/pip-s3cmd.png b/docs/data_management/storage/lts/images/pip-s3cmd.png similarity index 100% rename from docs/data_management/lts/images/pip-s3cmd.png rename to docs/data_management/storage/lts/images/pip-s3cmd.png diff --git a/docs/data_management/lts/images/policy.png b/docs/data_management/storage/lts/images/policy.png similarity index 100% rename from docs/data_management/lts/images/policy.png rename to docs/data_management/storage/lts/images/policy.png diff --git a/docs/data_management/lts/images/s3cfg.png b/docs/data_management/storage/lts/images/s3cfg.png similarity index 100% rename from docs/data_management/lts/images/s3cfg.png rename to docs/data_management/storage/lts/images/s3cfg.png diff --git a/docs/data_management/lts/images/shared-lts-config.png b/docs/data_management/storage/lts/images/shared-lts-config.png similarity index 100% rename from docs/data_management/lts/images/shared-lts-config.png rename to docs/data_management/storage/lts/images/shared-lts-config.png diff --git a/docs/data_management/lts/images/simplified-lts-core-diagram.png b/docs/data_management/storage/lts/images/simplified-lts-core-diagram.png similarity index 100% rename from docs/data_management/lts/images/simplified-lts-core-diagram.png rename to docs/data_management/storage/lts/images/simplified-lts-core-diagram.png diff --git a/docs/data_management/lts/index.md b/docs/data_management/storage/lts/index.md similarity index 89% rename from docs/data_management/lts/index.md rename to docs/data_management/storage/lts/index.md index f154a1d31..5808a885d 100644 --- a/docs/data_management/lts/index.md +++ b/docs/data_management/storage/lts/index.md @@ -5,7 +5,7 @@ UAB Long-term storage (LTS) is an S3 object-storage platform hosted at UAB. This !!! tip - [Globus](../transfer/globus.md#long-term-storage-s3-lts-connector) may be used to transfer data with LTS. + [Globus](../../transfer/globus/globus_individual_tutorial.md#long-term-storage-s3-lts-connector) may be used to transfer data with LTS. ## Terminology @@ -24,13 +24,13 @@ When talking about S3 storage, some terms are different compared to a normal fil Never give access and secret keys for individual or shared allocations to anyone! Treat them as usename and password, respectively. - If you need to give elevated permissions to other users to view, upload, download, delete, etc. any data from a bucket, those permissions can be changed via [bucket policies](iam_and_policies.md) without giving out keys. Please [contact Research Computing](../../index.md#how-to-contact-us) for help setting up and applying policies if you need it + If you need to give elevated permissions to other users to view, upload, download, delete, etc. any data from a bucket, those permissions can be changed via [bucket policies](iam_and_policies.md) without giving out keys. Please [contact Research Computing](../../../index.md#how-to-contact-us) for help setting up and applying policies if you need it !!! note - If you lose your access and secret keys, please submit a [support ticket](../../index.md#how-to-contact-us) to to request your keys. Keys will only be given to an allocation owner as verified by RC staff. + If you lose your access and secret keys, please submit a [support ticket](../../../index.md#how-to-contact-us) to to request your keys. Keys will only be given to an allocation owner as verified by RC staff. This documentation will use the standard file and path terms since those are more easily understood by most users. Just be aware that documentation such as [AWS CLI](https://docs.aws.amazon.com/cli/latest/) will use terms prefix, object, and others that are not standard in a typical filesystem. @@ -49,4 +49,4 @@ Better practice when naming buckets is to use a short, descriptive and memorable ## What to Do With Data When I Leave UAB? -See our [Leaving UAB page](../../account_management/leaving_uab.md). +See our [Leaving UAB page](../../../account_management/leaving_uab.md). diff --git a/docs/data_management/lts/interfaces.md b/docs/data_management/storage/lts/interfaces.md similarity index 90% rename from docs/data_management/lts/interfaces.md rename to docs/data_management/storage/lts/interfaces.md index a73ce1c5e..aff555523 100644 --- a/docs/data_management/lts/interfaces.md +++ b/docs/data_management/storage/lts/interfaces.md @@ -8,7 +8,7 @@ LTS is not available as a mounted filesystem on local computers or Cheaha. You m ## Globus -[Globus](../transfer/globus.md#long-term-storage-s3-lts-connector) is a general file transfer system that operates through a web browser and is recommended for most file transfer needs. UAB has an S3 connector for Globus that can transfer data to and from LTS as long as the user has access to the desired buckets. +[Globus](../../transfer/globus/globus_individual_tutorial.md#long-term-storage-s3-lts-connector) is a general file transfer system that operates through a web browser and is recommended for most file transfer needs. UAB has an S3 connector for Globus that can transfer data to and from LTS as long as the user has access to the desired buckets. To connect to the LTS endpoint in Globus, search `UAB Research Computing LTS` in the search bar and enter your access and secret keys given to you by Research Computing staff. You will be able to see the buckets owned by the allocation associated with the keys you entered. @@ -24,7 +24,7 @@ Globus is very useful for single transfers of data either to or from LTS and is ### Managing LTS Credentials on Globus -See our [Globus - Adding LTS Allocation Credentials](../transfer/globus.md#adding-lts-allocation-credentials) section for more information. +See our [Globus - Adding LTS Allocation Credentials](../../transfer/globus/globus_individual_tutorial.md#adding-lts-allocation-credentials) section for more information. ## Command Line @@ -35,7 +35,7 @@ While globus is the recommended tool for most data transfers, command line tools ### Installation of `s3cmd` and `s5cmd` on Cheaha -To install the tools on Cheaha, you can request a compute node through Cheaha's [Open OnDemand web portal](../../cheaha/open_ondemand/ood_layout.md#creating-an-interactive-job).Once your job is launched, open a terminal to execute the commands listed below. You do not need to install both tools if they aren't necessary. Both are available to install into [Anaconda](../../workflow_solutions/using_anaconda.md) environments. It's suggested to create a single environment named `s3` and install both s3cmd and s5cmd into it for easy access to both tools. Specific install and usage commands for each are given in their respective sections. You can create the general environment using the following commands: +To install the tools on Cheaha, you can request a compute node through Cheaha's [Open OnDemand web portal](../../../cheaha/open_ondemand/ood_layout.md#creating-an-interactive-job).Once your job is launched, open a terminal to execute the commands listed below. You do not need to install both tools if they aren't necessary. Both are available to install into [Anaconda](../../../workflow_solutions/using_anaconda.md) environments. It's suggested to create a single environment named `s3` and install both s3cmd and s5cmd into it for easy access to both tools. Specific install and usage commands for each are given in their respective sections. You can create the general environment using the following commands: ``` bash module load Anaconda3 @@ -261,7 +261,7 @@ Replace `` with the profile name defined for you ### Installation of `s3cmd` and `s5cmd` on Individual Systems Without Anaconda -The installation instructions and software dependencies may differ depending on the operating system being used. Following are the installation instructions tested for different operating systems. You may also use [Anaconda](../../workflow_solutions/using_anaconda.md) to install either or both packages. +The installation instructions and software dependencies may differ depending on the operating system being used. Following are the installation instructions tested for different operating systems. You may also use [Anaconda](../../../workflow_solutions/using_anaconda.md) to install either or both packages. #### Ubuntu @@ -272,7 +272,7 @@ sudo apt update sudp apt install s3cmd ``` -To install `s5cmd`, you will have to first install `go` software version `>=1.19`. Please refer to the [Download page](https://go.dev/dl/), and [installation instruction](https://go.dev/doc/install) for further details. After intalling `go`, you can build `s5cmd` in your `$HOME` directory using the below steps. +To install `s5cmd`, you will have to first install `go` software version `>=1.19`. Please refer to the [Download page](https://go.dev/dl/), and [installation instruction](https://go.dev/doc/install) for further details. After installing `go`, you can build `s5cmd` in your `$HOME` directory using the below steps. ```bash cd $HOME @@ -303,10 +303,10 @@ brew install gnupg #### Windows -To install `s3cmd` and `s5cmd` on a Windows system, you will first need to install [Windows Subsystem for Linux (WSL)](../../uab_cloud/remote_access.md#windows-subsystem-for-linux-wsl). Once WSL is installed, you can use the command line instructions for [Ubuntu](#ubuntu) to install `s3cmd` and `s5cmd`. +To install `s3cmd` and `s5cmd` on a Windows system, you will first need to install [Windows Subsystem for Linux (WSL)](../../../uab_cloud/remote_access.md#windows-subsystem-for-linux-wsl). Once WSL is installed, you can use the command line instructions for [Ubuntu](#ubuntu) to install `s3cmd` and `s5cmd`. For more information on `s3cmd` and `s5cmd`, please refer to the official [s3tools Page](https://s3tools.org/download), and [s5cmd page](https://github.com/peak/s5cmd?tab=readme-ov-file). ## Alternatives -There are other tools for interfacing with LTS such as rclone. Please see our [rclone documentation](../transfer/rclone.md) for more details. +There are other tools for interfacing with LTS such as rclone. Please see our [rclone documentation](../../transfer/rclone.md) for more details. diff --git a/docs/data_management/lts/lts_cores.md b/docs/data_management/storage/lts/lts_cores.md similarity index 99% rename from docs/data_management/lts/lts_cores.md rename to docs/data_management/storage/lts/lts_cores.md index 73673d55e..077e08add 100644 --- a/docs/data_management/lts/lts_cores.md +++ b/docs/data_management/storage/lts/lts_cores.md @@ -42,7 +42,7 @@ The details concerning data transfer from Core instruments and analysis machines Do not set up transfers from machines that Core customers will have access to. This creates opportunities for researchers to access data they are not authorized for. Data collected by the Core should only be accessible by Core personnel until those data are distributed to their respective labs. -For situations where data either needs to be transferred from multiple machines to LTS. please contact [Research Computing](../../index.md#how-to-contact-us) for a consultation. +For situations where data either needs to be transferred from multiple machines to LTS. please contact [Research Computing](../../../index.md#how-to-contact-us) for a consultation. ## Distributing to Data Owners diff --git a/docs/data_management/lts/lts_faq.md b/docs/data_management/storage/lts/lts_faq.md similarity index 97% rename from docs/data_management/lts/lts_faq.md rename to docs/data_management/storage/lts/lts_faq.md index 1fa5e9f63..34a07cccd 100644 --- a/docs/data_management/lts/lts_faq.md +++ b/docs/data_management/storage/lts/lts_faq.md @@ -31,7 +31,7 @@ While these are two simple solutions, a combination of both can be implemented w ## Are Automatic Backups to LTS Available? -Automatic backups are not available by default. If you would like to periodically sync your bucket to a directory on your local machine or Cheaha, you will need to set up a cron task to submit a Slurm job that will run a sync. IF you would like to implement this for your own bucket, please [contact us](../../index.md#how-to-contact-us). +Automatic backups are not available by default. If you would like to periodically sync your bucket to a directory on your local machine or Cheaha, you will need to set up a cron task to submit a Slurm job that will run a sync. IF you would like to implement this for your own bucket, please [contact us](../../../index.md#how-to-contact-us). ## Why Can I Not Interact With a File in My Bucket? @@ -125,4 +125,4 @@ The following policy file will give read permission to all LTS users for all obj ## Can I Change Permissions on a Bucket via Globus? -As of now, there is no way to change permissions on a bucket via [Globus](../transfer/globus.md). The only way to change permissions is via the command line. +As of now, there is no way to change permissions on a bucket via [Globus](../../transfer/globus/index.md). The only way to change permissions is via the command line. diff --git a/docs/data_management/lts/res/example-active-external-storage-policy.json b/docs/data_management/storage/lts/res/example-active-external-storage-policy.json similarity index 100% rename from docs/data_management/lts/res/example-active-external-storage-policy.json rename to docs/data_management/storage/lts/res/example-active-external-storage-policy.json diff --git a/docs/data_management/lts/res/example-synced-project-policy.json b/docs/data_management/storage/lts/res/example-synced-project-policy.json similarity index 100% rename from docs/data_management/lts/res/example-synced-project-policy.json rename to docs/data_management/storage/lts/res/example-synced-project-policy.json diff --git a/docs/data_management/lts/tutorial/index.md b/docs/data_management/storage/lts/tutorial/index.md similarity index 100% rename from docs/data_management/lts/tutorial/index.md rename to docs/data_management/storage/lts/tutorial/index.md diff --git a/docs/data_management/lts/tutorial/individual_lts_tutorial.md b/docs/data_management/storage/lts/tutorial/individual_lts_tutorial.md similarity index 93% rename from docs/data_management/lts/tutorial/individual_lts_tutorial.md rename to docs/data_management/storage/lts/tutorial/individual_lts_tutorial.md index 3bb2ce38b..868ca2cf6 100644 --- a/docs/data_management/lts/tutorial/individual_lts_tutorial.md +++ b/docs/data_management/storage/lts/tutorial/individual_lts_tutorial.md @@ -7,21 +7,21 @@ In this tutorial, we will guide you through using `s3cmd` on the Cheaha system t ## Prerequisites -To get up to speed, you should have a basic understanding of how to use the shell/terminal. If you’re not familiar with these concepts, we recommend checking out our [learning resources on basic shell usage](../../../workflow_solutions/shell.md#shell-reference). +To get up to speed, you should have a basic understanding of how to use the shell/terminal. If you’re not familiar with these concepts, we recommend checking out our [learning resources on basic shell usage](../../../../workflow_solutions/shell.md#shell-reference). -You will also need an individual LTS allocation created by our team. If you believe you need an allocation but do not have one, please [contact us](../../../index.md#how-to-contact-us). +You will also need an individual LTS allocation created by our team. If you believe you need an allocation but do not have one, please [contact us](../../../../index.md#how-to-contact-us). ## Setting Up Your Environment ### Install `s3cmd` Within Conda Environment on Cheaha -To interact with LTS (Long-Term Storage) using [S3 (Simple Storage Service)](https://aws.amazon.com/s3/), you need the `s3cmd` tool installed.[`s3cmd`](https://s3tools.org/s3cmd) is a command-line tool for managing files in cloud storage systems like S3. It's recommended to install it using `pip`, the standard package installer for Python, which allows you to install packages from the [Python Package Index (PyPI)](https://pypi.org/), within a [Conda environment](../../../workflow_solutions/using_anaconda.md#create-an-environment) on Cheaha. +To interact with LTS (Long-Term Storage) using [S3 (Simple Storage Service)](https://aws.amazon.com/s3/), you need the `s3cmd` tool installed.[`s3cmd`](https://s3tools.org/s3cmd) is a command-line tool for managing files in cloud storage systems like S3. It's recommended to install it using `pip`, the standard package installer for Python, which allows you to install packages from the [Python Package Index (PyPI)](https://pypi.org/), within a [Conda environment](../../../../workflow_solutions/using_anaconda.md#create-an-environment) on Cheaha. Please avoid using `conda install s3cmd`, as that version will not work as expected. Instead, follow the steps below to install `s3cmd` using `pip` within your Conda environment. -First, access our interactive Open OnDemand (OOD) portal at and create a job on Cheaha using one of our interactive applications. For guidance, refer to our tutorial on [installing and setting Conda environment](../../../cheaha/tutorial/pytorch_tensorflow.md#installing-anaconda-environments-using-the-terminal). +First, access our interactive Open OnDemand (OOD) portal at and create a job on Cheaha using one of our interactive applications. For guidance, refer to our tutorial on [installing and setting Conda environment](../../../../cheaha/tutorial/pytorch_tensorflow.md#installing-anaconda-environments-using-the-terminal). -Once your interactive apps session is launched, open the terminal as described in [step 5 of the Anaconda tutorial page](../../../cheaha/tutorial/pytorch_tensorflow.md#installing-anaconda-environments-using-the-terminal) and run the below commands. +Once your interactive apps session is launched, open the terminal as described in [step 5 of the Anaconda tutorial page](../../../../cheaha/tutorial/pytorch_tensorflow.md#installing-anaconda-environments-using-the-terminal) and run the below commands. ```bash module load Anaconda3 @@ -36,7 +36,7 @@ Once these steps are completed, verify the installation by running `pip list | g ### Install `s3cmd` on Your Local Systems -To install s3cmd on your local machine, please follow the instructions provided in [our s3cmd documentation for local installation](../../../data_management/lts/interfaces.md#installation-of-s3cmd-and-s5cmd-on-individual-systems-without-anaconda). +To install s3cmd on your local machine, please follow the instructions provided in [our s3cmd documentation for local installation](../interfaces.md#installation-of-s3cmd-and-s5cmd-on-individual-systems-without-anaconda). ### Configuring `s3cmd` for LTS Buckets @@ -52,7 +52,7 @@ This will prompt you to enter the access key and secret key associated with your ![image-s3cmd](../images/config-s3cmd.png) -Once the configuration is complete, `s3cmd` will generate a `.s3cfg` file in your home directory (`$HOME`), as shown below. To find your home directory in Cheaha and view the `.s3cfg` file, follow the instructions on our [Navigating Open OnDemand](../../../cheaha/open_ondemand/ood_layout.md#navigating-open-ondemand) page. Be sure to check the **Show Dotfiles** option in the top right corner to make hidden files visible. +Once the configuration is complete, `s3cmd` will generate a `.s3cfg` file in your home directory (`$HOME`), as shown below. To find your home directory in Cheaha and view the `.s3cfg` file, follow the instructions on our [Navigating Open OnDemand](../../../../cheaha/open_ondemand/ood_layout.md#navigating-open-ondemand) page. Be sure to check the **Show Dotfiles** option in the top right corner to make hidden files visible. ![config-file](../images/s3cfg.png) diff --git a/docs/data_management/lts/tutorial/shared_lts_tutorial.md b/docs/data_management/storage/lts/tutorial/shared_lts_tutorial.md similarity index 94% rename from docs/data_management/lts/tutorial/shared_lts_tutorial.md rename to docs/data_management/storage/lts/tutorial/shared_lts_tutorial.md index 5a7467905..fe5bb661a 100644 --- a/docs/data_management/lts/tutorial/shared_lts_tutorial.md +++ b/docs/data_management/storage/lts/tutorial/shared_lts_tutorial.md @@ -27,7 +27,7 @@ The screenshot below shows the full terminal output of `s3cmd --configure -c $HO Once the configuration is complete, `s3cmd` will generate a file in your home directory `$HOME` using the profile name you provided. For example, if you name your shared LTS profile `my-lab`, the configuration file will be saved as `$HOME/my-lab`. This will be in addition to the default `.s3cfg` file that was previously configured for your individual LTS, as shown below. -The screenshot displays a window listing files in the user's home directory. Among the listed files, `.s3cfg` appears alongside other dotfiles, and `my-lab` appears as a separate, non-hidden file. These entries confirm the presence of both configuration files created by `s3cmd`. To locate your home directory on Cheaha and view both the `.s3cfg` and `my-lab`files, follow the instructions on our [Navigating Open OnDemand](../../../cheaha/open_ondemand/ood_layout.md#navigating-open-ondemand) page. Make sure to enable the "Show Dotfiles" option in the top-right corner of the Open OnDemand (OOD) file browser to make hidden files visible. +The screenshot displays a window listing files in the user's home directory. Among the listed files, `.s3cfg` appears alongside other dotfiles, and `my-lab` appears as a separate, non-hidden file. These entries confirm the presence of both configuration files created by `s3cmd`. To locate your home directory on Cheaha and view both the `.s3cfg` and `my-lab`files, follow the instructions on our [Navigating Open OnDemand](../../../../cheaha/open_ondemand/ood_layout.md#navigating-open-ondemand) page. Make sure to enable the "Show Dotfiles" option in the top-right corner of the Open OnDemand (OOD) file browser to make hidden files visible. ![s3cmd config files for shared and individual LTS profiles shown in your home directory in the OOD file browser.](../images/my-lab.png) @@ -47,4 +47,4 @@ Here are examples for creating and listing buckets, and apply bucket policies: To manage your shared LTS buckets, repeat what you did with your [individual buckets](./individual_lts_tutorial.md#managing-buckets), and add the `-c my-lab` option. -If you have any questions about configuring LTS, switching between LTS allocations, or managing bucket access across multiple LTS profiles, please [Contact Us](../../../index.md#how-to-contact-us). +If you have any questions about configuring LTS, switching between LTS allocations, or managing bucket access across multiple LTS profiles, please [Contact Us](../../../../index.md#how-to-contact-us). diff --git a/docs/data_management/res/cheaha_individual_account.csv b/docs/data_management/storage/res/cheaha_individual_account.csv similarity index 100% rename from docs/data_management/res/cheaha_individual_account.csv rename to docs/data_management/storage/res/cheaha_individual_account.csv diff --git a/docs/data_management/res/cheaha_project_directory.csv b/docs/data_management/storage/res/cheaha_project_directory.csv similarity index 100% rename from docs/data_management/res/cheaha_project_directory.csv rename to docs/data_management/storage/res/cheaha_project_directory.csv diff --git a/docs/data_management/res/individual_allocation_functional_roles.csv b/docs/data_management/storage/res/individual_allocation_functional_roles.csv similarity index 100% rename from docs/data_management/res/individual_allocation_functional_roles.csv rename to docs/data_management/storage/res/individual_allocation_functional_roles.csv diff --git a/docs/data_management/res/project_space_permissions.csv b/docs/data_management/storage/res/project_space_permissions.csv similarity index 100% rename from docs/data_management/res/project_space_permissions.csv rename to docs/data_management/storage/res/project_space_permissions.csv diff --git a/docs/data_management/res/shared_allocation_functional_roles.csv b/docs/data_management/storage/res/shared_allocation_functional_roles.csv similarity index 100% rename from docs/data_management/res/shared_allocation_functional_roles.csv rename to docs/data_management/storage/res/shared_allocation_functional_roles.csv diff --git a/docs/data_management/res/storage_overview.csv b/docs/data_management/storage/res/storage_overview.csv similarity index 88% rename from docs/data_management/res/storage_overview.csv rename to docs/data_management/storage/res/storage_overview.csv index e05371df0..6da1ad269 100644 --- a/docs/data_management/res/storage_overview.csv +++ b/docs/data_management/storage/res/storage_overview.csv @@ -7,5 +7,5 @@ Cheaha Path,No path; see [LTS Interfaces](lts/interfaces.md).,`/home/$USER` (`$H Read/Write (IO) Speed,Slower,Fast,Fast,Fast,Fastest Redundancy,Erasure Coding (9+3),RAID 6,RAID 6,RAID 6,N/A (Ephermal; data may be deleted anytime) Responsibilities & Procedures,PI/Director responsible for data and access control.,,PI/Director responsible for data and access control.,Data deleted after 30 days.,Data deleted as needed. -Access Control,[Bucket policies](./lts/iam_and_policies.md),Self only,[`chmod`](../workflow_solutions/shell.md#manage-permissions-of-files-and-directores-chmod) and [ACLs](../workflow_solutions/shell.md#manage-researcher-access-to-files-and-directories-getfacl-setfacl),Self only,Self only +Access Control,[Bucket policies](./lts/iam_and_policies.md),Self only,[`chmod`](../../workflow_solutions/shell.md#manage-permissions-of-files-and-directores-chmod) and [ACLs](../../workflow_solutions/shell.md#manage-researcher-access-to-files-and-directories-getfacl-setfacl),Self only,Self only How to Request,Upon request.,Comes with Cheaha account.,Upon request.,Comes with Cheaha account.,Comes with Cheaha account. diff --git a/docs/data_management/transfer/filezilla.md b/docs/data_management/transfer/filezilla.md index 1cb21ce5f..2187e7fe5 100644 --- a/docs/data_management/transfer/filezilla.md +++ b/docs/data_management/transfer/filezilla.md @@ -29,7 +29,7 @@ Click `Quickconnect` and you will be connected to the remote storage system. The When connecting in the future, you will be able to select the connection from the dropdown arrow next to the Quickconnect button. -In both the local and remote panes, you can navigate to the directories you are transferring from and to. You only have access to directories you can normally access on Cheaha, so your [Individual Storage](../index.md#what-individual-storage-solutions-are-available) as well as any [Project Storage](../index.md#what-shared-storage-solutions-are-available) directories you have been added to. +In both the local and remote panes, you can navigate to the directories you are transferring from and to. You only have access to directories you can normally access on Cheaha, so your [Individual Storage](../storage/index.md#what-individual-storage-solutions-are-available) as well as any [Project Storage](../storage/index.md#what-shared-storage-solutions-are-available) directories you have been added to. ### Transferring Data diff --git a/docs/data_management/transfer/globus.md b/docs/data_management/transfer/globus.md deleted file mode 100644 index 80e63c2a4..000000000 --- a/docs/data_management/transfer/globus.md +++ /dev/null @@ -1,342 +0,0 @@ -# Globus - - -!!! announcement - - We are aware of reduced transfer speed into and out of [Long-Term Storage (LTS)] when using Globus and are working on a solution. For some use cases, [`s5cmd`](../lts/interfaces.md#s5cmd) may be a faster alternative. Please see [Our News Section](../../news/posts/2025-04-07-reduced-lts-transfer-speeds-on-globus.md) for more information. - - -Globus is a powerful tool for robustly and securely managing data transfers to and from collaborators and within UAB Research Computing. Globus is recommended for most single-use, day-to-day data transfer use-cases. - -UAB Research Computing uses High Assurance Endpoints, meaning there are additional security measures in place to reduce risk and move toward HIPAA compliance. Generally speaking, if you have used Globus in the past, the data transfer interface has not changed, but there are a few new restrictions/changes. - -1. You will be prompted to prove authorization each time you access a UAB Research Computing endpoint or attempt to download files to your local machine from such an endpoint. If you are already logged in with Single Sign-On (SSO) the process is simple. If not, you will need to authenticate with SSO. -1. Bookmarks are not allowed in High Assurance endpoints. -1. In the newer version, Globus "Endpoints" have been moved under "Console", which can be located in the left-hand navigation pane. - ![!Globus endpoints moved under console](./images/globus_108_console_endpoint.png) - -For more detailed information on High Assurance please see the Globus official pages below: - -- [High Assurance Security Overview](https://docs.globus.org/guides/overviews/security/high-assurance-overview/) -- [High Assurance Collections](https://docs.globus.org/guides/overviews/high-assurance/) - -## Setting Up Globus Connect Personal - -[Globus Connect Personal](https://www.globus.org/globus-connect-personal) is software meant to be installed on local machines such as laptops, desktops, -workstations and self-owned, local-scale servers. Globus maintains excellent documentation for installation on [MacOS](https://docs.globus.org/globus-connect-personal/install/mac/), [Linux](https://docs.globus.org/globus-connect-personal/install/linux/) and [Windows](https://docs.globus.org/globus-connect-personal/install/windows/). - -To verify your installation is complete, please visit and log in. Click "Collections" in the left-hand navigation pane and then click the "Administered By You" tab. Look in the table for the Guest Collection you just created. - -## Managing Identities - -Globus Identities is a concept helping to map Globus Accounts (one per person) to institutions (one or more per person). - -Most UAB researchers will have a single identity, their UAB identity, tied to their BlazerID. Some researchers may have external collaborations or appointments that provide additional entities which need access to other endpoints on Globus. - -To manage your identities, navigate to and sign in. - - -!!! important - - To use UAB Research Computing endpoints, you will need to ensure you are using your UAB identity. - - -## Moving Data Between Collections - -1. Log in to the Globus App online at using UAB Single Sign-On (SSO). Start typing "University of Alabama at Birmingham" into the "Use your existing organizational login" text box and selected it when it appears in the list. - - ![!Globus login page with University of Alabama at Birmingham entered into the text box](./images/globus_001_login.png) - -1. Click File Manager in the left-hand navigation pane. - - ![!Navigation pane with File Manager selected.](./images/globus_002_nav_pane_file_manager.png) - -1. Ensure the center icon for the "Panels" selection is picked. - - ![!Panels selection widget with center icon selected. Center icon appears to be two side-by-side panes.](./images/globus_003_panels.png) - -1. Click the "Search" icon in the "Collection" text box near the top-left or top-right of the page to locate an endpoint/collection. An endpoint is a server that provides access to data stored in a cluster. A Collection can be a Mapped Collection or Guest Collection, which represents a user's own collection of files or directories mapped from storage endpoints to their Globus account for easy management, sharing and accessing. Please refer to the [Globus endpoints and collections](https://docs.globus.org/guides/overviews/collections-and-endpoints/) to know more. There are multiple ways to find an endpoint. For some endpoints you may be asked to log in, which is true of all UAB endpoints. Some UAB endpoints may also require that you be on the UAB Campus VPN. - - ![!Globus File Manager interface with mouse pointer over left-hand Collection Search box.](./images/globus_004_search_bar.png) - - 1. Begin typing in the box to search for an endpoint. To find UAB-related endpoints, search for "UAB". There are two Cheaha endpoints - - 1. Cheaha cluster on-campus (UAB Science DMZ) for machines that are either on the UAB Campus Network, or connected to the UAB Campus VPN. - 1. Cheaha cluster off-campus (UAB Science DMZ) for machines that are _not_ on the UAB Campus Network and _not_ on the UAB Campus VPN. - - 1. The "Recent" tab shows endpoints/collections that have most recently been used. - - ![!Globus Collection Search Recent tab showing two endpoints.](./images/globus_005_recent_tab.png) - - 1. The "Bookmarks" tab shows a list of collection bookmarks. Bookmarks may not reference folders within UAB Research Computing or other High Assurance endpoints. - - ![!Globus Collection Search Bookmarks tab showing four bookmarks.](./images/globus_006_bookmarks_tab.png) - - 1. The "Your Collections" tab shows all collections owned by you. For most researchers this will be one or more Globus Connect Personal collections. - - ![!Globus Collection Search Your Collections tab showing one endpoint.](./images/globus_007_your_collections_tab.png) - - 1. The "Shared With You" tab shows any private collections that have been shared with you by other users, possibly collaborators. - -1. When an endpoint/collection has been selected you will see a list of folders and files on the default path for that endpoint/collection in the bottom box. You can use the "Path" box to type a path to find the files you are looking for. - - ![!Globus File Manager interface with one endpoint selected showing files of default directory.](./images/globus_010_one_endpoint_done.png) - -1. Repeat the process of selecting an endpoint/collection for the other "Collection" text box. - - ![!Globus File Manager interface with both endpoints selected showing files for both default directories.](./images/globus_011_two_endpoint_done.png) - -1. When both endpoints have been selected and you have chosen the correct paths for each collection, select files and/or folders on the side you wish to transfer FROM. We will call this side the source collection, and the other side the target collection. Selections may be made by clicking the checkboxes that appear when you hover over each file or folder. - - ![!Globus File Manager interface with files selected in left endpoint.](./images/globus_012_selected_files.png) - -1. When all files and folders have been selected from the source collection, click the "Start" button on the source collection side. This will start a transfer process from source to target. The files will be placed in the currently open path on the target collection. - - ![!Pop-up showing Transfer request submitted successfully. Pop-up contains link to View details.](./images/globus_013_popup.png) - -1. A green pop-up notification will appear indicating the transfer has started. Click "View details \>" to be taken to the status of the transfer. You can also check on the status of any transfers by clicking the "Activity" button in the left-hand navigation pane. - - -!!! note - - File permissions from the source will not be copied to the destination. Please read more at [this ask.ci FAQ](https://ask.cyberinfrastructure.org/t/why-cant-other-users-access-data-i-transferred-to-a-project-space-on-cheaha/2527/3). - - -### Transfer and Sync Options - -Between the two "Start" buttons on the "File Manager" page is a "Transfer & Timer Options" drop down menu. Click that button to change the options. More information on each option. A brief summary of the options under "Transfer Settings" are... - -![!Transfer and Timer Options pane showing multiple options.](./images/globus_040_transfer_and_sync_options.png) - -1. sync - Sync files only, rather than create new files. -1. delete files - Delete any files on the target that are not on the source. Useful for forcing identical filesystems when syncing. -1. preserve source - Copies file "modified time" metadata. -1. verify integrity - Verifies that checksums are identical on source and target after transfer completes. The default option is to verify integrity. Its highly recommended to leave this unchecked. -1. encrypt transfer - Encrypts data before leaving source and decrypts after arriving at destination. Recommended for all transfers, required and enforced for all UAB endpoints. It is checked by default. -1. skip files - Skips source files that cause errors during the transfer. Otherwise the entire transfer will stop when an error is encountered. -1. quota fail - Fails instead of retries when the target storage quota is exceeded. -1. Apply filter - Filter rules can be used to customize and fine-tune the transfer process, ensuring that only the desired files/directories are transferred while excluding others based on the rules defined. - -Below the "Transfer Settings", there are additional options available that you can enable to customize notification of success/failure, and schedule the transfer to occur at specific time. - -### Common Errors - -1. File Not Found - This may mean that a file was not readable by Globus. Check that the file hasn't moved or changed names during the transfer. It is recommended to not modify files while they are being transferred by Globus. -1. Permission Denied - Globus is not able to access the files because permissions do not allow it. For Globus Connect Personal, be sure the containing folder is on the "Accessible Folders" list. Be sure that your Cheaha account has access to read the file. - -### Project Space Permissions - -Globus does not preserve permissions nor ownership when data is transferred, instead using whatever permissions are default at the target location, and making the owner the authenticated user who initiated the transfer. Typically this is not an issue, but may cause problems for [Project Storage directories](../index.md#what-shared-storage-solutions-are-available). Please see our [Project Directory Permissions Section](../cheaha_storage_gpfs/project_directories.md#project-directory-permissions) for more information. - -### More Information - -A [Globus FAQ](https://docs.globus.org/faq/globus-connect-endpoints/) is available for additional information on endpoints and transfers. - -## Connectors - -UAB Researcher Computing has subscriptions to connectors for cloud services and other types of filesystems. - -### UAB Box Connector - -To use the UAB Box Connector, [search for a collection/endpoint](#moving-data-between-collections) like usual and enter "UAB Box" into the search box. Select the endpoint labeled "UAB Box". You should see a list of files and folders that are available to you at . File transfers work as they would with any other endpoint or collection. - -### Long-Term Storage S3 (LTS) Connector - - -!!! important - - [LTS](../lts/index.md) behaves differently from other file systems and comes with a few possible pitfalls. Keep in mind the following three rules: (1) all data must be in buckets, (2) buckets are only allowed in the root folder, and (3) buckets must have unique names. - - -To use the UAB [LTS](../lts/index.md) Connector, [search for a collection/endpoint](#moving-data-between-collections) like usual and enter "UAB LTS" into the search box. Select the endpoint labeled "UAB Research Computing LTS (Long Term Storage aka S3)". If you have stored data within LTS already you should see a list of folders, otherwise you will see an empty space where folders may be placed. Each folder corresponds to a [bucket](../lts/index.md#terminology) in LTS. To create a bucket, click "New Folder" in the "File Manager" window in Globus. Note that buckets must have globally unique names. Read on for more information about possible pitfalls. - - ![create bucket via globus](./images/globus-create-lts-bucket.png) - -#### Adding LTS Allocation Credentials - -Before you can manage an LTS allocation using Globus, you'll need to add the [LTS S3 Access and Secret Keys](../lts/index.md#requesting-an-allocation) to the LTS endpoint. To do so, please follow the instructions given below. - -1. In your browser, navigate to and login using UAB SSO. -1. Click "Collections" in the left-hand navigation pane. -1. In the search field at the top of the "Collection" page, enter "UAB LTS" and then click the search button. Be sure to uncheck "Recently Used" if it is checked. You should see a collection titled "UAB Research Computing LTS (Long Term Storage aka S3)" appear in the search results. -1. Click the right arrow indicator at the right-hand side of the "UAB Research Computing LTS" result. You should be taken to the Overview page for the "UAB Research Computing LTS" endpoint. - - ![!Image showing Globus collections page with UAB LTS in search box, UAB Research Computing LTS endpoint visible in results, and right arrow highlighted.](./images/globus-lts-s3-credentials-01.png) - -1. Near the top of the Overview page, click the "Credentials" tab". After some time, you should see a form requesting your "AWS IAM Access Key ID" and "AWS IAM Secret Key". -1. Enter the two requested keys in the appropriate boxes, then press "Continue". For data security reasons, you may be asked to authenticate again with your UAB SSO credentials. If you have more than one set of keys, you will need to choose which to enter. At this time, there doesn't appear to be a way to enter or use more than one set of credentials at a time with Globus. - - ![!Image showing Register a Credential form on the UAB Research Computing LTS endpoint Overview page showing artificial keys entered into the appropriate text fields.](./images/globus-lts-s3-credentials-02.png) - -1. You should be taken back to the UAB Research Computing LTS endpoint Overview page with the Credentials tab selected. There should now be one entry on this page showing your AWS IAM Access Key ID and your Globus Identity (i.e., your `BlazerID@uab.edu` email address). If you have multiple Access Keys and ever wish to change which one is being used with Globus, then click the "Trash Can" icon next to the entry on this page to delete it, and start the key entry process over from the previous step. - - ![!Image showing UAB Research Computing LTS endpoint Overview page showing Access Credentials entry with AWS IAM Access Key ID and Globus Identity.](./images/globus-lts-s3-credentials-03.png) - -At this point you are able to access the LTS allocation associated with the Access Key you entered, and any buckets which have granted access to that Access Key. If you have more than one Access Key, e.g. for each of your personal and project allocations, you will need to choose which key to enter above. If you ever wish to change credentials, then click the "Trash Can" icon next to the entry shown in step 6, above, and start the key entry process over. - -#### Data Must Be in Buckets - -All data transferred to LTS must be placed in a bucket, and may _not_ be placed directly into the root directory. Attempting to move data to the root directory will result in an unhelpful error message in the "Activity" window. - -![!unhelpful error message for data placed in the root LTS directory](images/globus_lts_no_bucket_error_001.png) - -Clicking on the "view event log" link shows the following. - -![!unhelpful event log message](images/globus_lts_no_bucket_error_002.png) - -```text -Error (transfer) -Endpoint: UAB Research Computing LTS (Long Term Storage aka S3) (184408b4-d04b-4513-9912-8feeb6adcab3) -Server: m-a201b5.9ad93.a567.data.globus.org:443 -Command: STOR /test.log -Message: The connection to the server was broken --- -Details: an end-of-file was reached\nglobus_xio: An end of file occurred\n -``` - -#### Buckets Must Have Globally Unique Names - -When creating new buckets, the name must be unique across all buckets on the system. If a duplicate bucket name, for example `first-test-bucket`, is entered, a long error message will appear in a small space next to the new bucket name. For readability, the expanded message is shown below. - -![!large error message in small space](images/globus_lts_duplicate_name_error_001.png) - -```text -Remote Endpoint Failure: Path already exists, Error (mkdir) -Endpoint: UAB Research Computing LTS (Long Term Storage aka S3) (184408b4-d04b-4513-9912-8feeb6adcab3) -Server: 138.26.220.68:443 -Message: Path '/first-test-bucket/' already exists -``` - -At first glance, requiring unique names across all buckets on the system may sound very restrictive, but it is necessary for LTS to be as fast as it is. Fortunately, there is an easy way to deal with the limitation. See our LTS section on [good naming practice](../lts/index.md#avoiding-duplicate-names-for-buckets) for how to avoid duplicate names. For example, if you want to name a bucket `ai-lab` for storing data related to the entire AI lab or a specific dataset, you can append a universally unique identifier (UUID) to the name. To generate a UUID, visit , and a 16-byte UUID will be automatically generated. You can then copy it and append it to the name `ai-lab`, as shown below. - -![!large error message in small space](images/globus-uuid.png) - -Similarly, if an invalid bucket name, such as `first_test_bucket`, is entered (due to the use of an underscore, which doesn’t follow LTS bucket naming rules), an error will also be displayed as shown below. To avoid this, please refer to the guidelines for [valid bucket name in LTS](../lts/lts_faq.md#what-are-valid-bucket-names-in-lts). - -![!large invalid error message in small space](images/globus_lts_invalid_name_error_001.png) - -```text -Bad Gateway: Endpoint Error, Error (mkdir) -Endpoint: UAB Research Computing LTS (Long Term Storage aka S3) (184408b4-d04b-4513-9912-8feeb6adcab3) -Server: 138.26.220.68:443 -Command: MKD /first_test_bucket/ -Message: Fatal FTP Response --- -Details: 500 -globus_gridftp_server_s3_base: S3 -Error accessing "": ErrorInvalidBucketName: ErrorInvalidBucketName: \r\n -``` - -To cancel or dismiss these errors, click the `refresh list` button on the Globus collection panel, or scroll all the way to the right in the red area of the error message and click the `x` symbol as shown below. - -![!large error message cancellation](images/globus_lts_cancele_name_error_001.png) - -Uploading a top-level folder that does not follow the bucket naming rules will cause an error similar to the one encountered when creating a bucket with an invalid name. When uploading a folder with a name similar to a bucket in your LTS, Globus will sync and save all sub-folders and files into that bucket. Subfolders also need to follow naming rules, but they inherit the uniqueness from the parent name. This is part of why we recommend random UUIDs. If you attempt to upload a folder with a name that matches an existing bucket in someone else's space, you will encounter a `permission denied error`, as shown below. - -![!large error message in small space](images/globus_lts_upload_error_001.png) - -Globus can create buckets. By default, buckets are created without a policy, meaning only you can access them until a policy is added. However, Globus cannot be used to modify or add policies. In addition, files transferred to a bucket will become objects with the same name, as long as the name is valid and not duplicated. Globus does not recognize or handle metadata, so you cannot use it to view or modify metadata. For guidance on defining policies for your bucket, please refer to our documentation on [policy structure](../lts/iam_and_policies.md#policy-structure) and [applying a policy](../lts/iam_and_policies.md#applying-a-policy). - -## Using Bookmarks - -To save a bookmark, use the File Manager interface to select a collection and navigate to a path on that collection. Then click the "Create Bookmarks" button as shown below. - -![!Globus File Manager interface with mouse pointer hovering over Bookmark icon.](./images/globus_060_create_bookmark.png) - -To manage bookmarks, click on the Collection search bar, then select the Bookmarks tab. To edit a bookmark, click the 'Pencil' icon. To delete a bookmark, click the 'Trash Bin' icon. - -![!Globus Bookmarks interface showing four bookmarks.](./images/globus_061_manage_bookmarks.png) - - -!!! note - - It is not possible to create bookmarks within High Assurance Endpoints. - - -## Managing Guest Collections From a Globus Connect Personal Endpoint - -It is NOT RECOMMENDED to make Globus Connect Personal Guest Collections public as this is insecure. It is more difficult to manage access controls for the entire Globus Connect Personal Collection than for a Guest Collection. Guest collections make it simpler to share different data with distinct collaborators, and to manage who has access to what data. Be secure, use Guest Collections! - -### Creating a Guest Collection - -1. Click "Collections" in the left-hand navigation pane. - -1. Click the "Administered By You" tab (or) you can search for the collections in search. - - ![!Globus Endpoints page with Administered by You selected, showing two endpoints. One of the endpoints is a shared endpoint.](./images/globus_100_shared_search_collections.png) - -1. In the table, find the collections you wish to share data from and click its name. You will be taken to the page for that collection. - -1. Click the "Collections" tab. - - ![!Globus UAB RC Work Laptop page with Guest Collections tab selected showing one collection.](./images/globus_101_shared_collections.png) - -1. Click the "Add a Guest Collection" button. - -1. Fill out the form. - - ![!Create New Guest Collection form.](./images/globus_102_shared_collection_form.png) - - 1. Manually enter a path or click the Browse button to select a folder. - 1. Give a short but memorable name for your Guest Collection. This information will be useful for your collaborators. - 1. Optionally fill in a more detailed description of the Guest Collection for your records. - 1. Optionally fill in searchable keywords. - 1. Other additional options include, information link, contact email, organization/department. - 1. Default directory, if left empty, is equivalent to the first field "Directory". - -1. Click "Create Collection" to move to the next step. You will be taken to the page for the newly created collection, which is now a full-fledged Guest Collection. Any further references to "an endpoint" will be about the newly created, Guest Collection. - -1. Make sure you are on the "Permissions" tab. You should see a permissions table with your name in it. - - ![!Newly created test endpoint page with Permissions tab selected.](./images/globus_103_shared_permissions.png) - -1. Click "Add Permissions -- Share With" to share your Guest Collection with other users. - -1. Fill out the form. - - ![!Test endpoint Add Permissions Share With form.](./images/globus_104_shared_add_permissions.png) - - 1. Optionally enter a path within the shared endpoint or use the Browse button. If you leave the path as just a slash, the entire shared endpoint will be shared with the permitted users. - 1. Select who to share with. - 1. User - One or more users. - 1. Group - All members of a group. - 1. All Users - All globus users. - 1. Public - Makes data accessible to everyone. - - - !!! danger - - It is important to note that options (iii) and (iv) poses a high risk in terms of security. Therefore, we strongly advise against this practice.This will expose information to everyone on Globus! - - - 1. Search for users to add, or a group, depending on your choice above. You should be able to find any globus user using the search box. - - - !!! warning - - Be certain of which user you are selecting! Check the email address domain. - - - 1. If adding users, optionally enter a message so they know why they are being added. - 1. Select permissions. Read is automatically selected and cannot be changed. Write permissions are optional. - -1. Click "Add Permission" to add permissions for these users or groups. You will be returned to the page for the Guest Collection and should be on the "Permissions" tab and should see the user or group in the table. - -### Deleting a Guest Collection - -1. Click "Collections" in the left-hand navigation pane, then - -1. Click the "Administered By You" tab. - - ![!Globus Endpoints page with Administered by You tab selected, showing two endpoints.](./images/globus_105_shared_collections.png) - -1. Click the right caret ">" icon at the right side of the row with the collection you wish to delete. You will be taken to the information page for that collection. - - ![!Globus Endpoints page with Shared Collection Overview.](./images/globus_106_shared_collection_overview.png) - -1. Click "X Delete Collection" and a confirmation dialog will open at the top of the page. Respond to the dialog to delete the Guest Collection, or to cancel. - - ![!Delete Collection confirmation dialog banner.](./images/globus_107_shared_delete.png) - -## Where Can I Learn More About Globus? - -- See our [Tutorials](./tutorial/index.md) -- Visit the [Globus Events Calendar](https://www.globus.org/events) for official learning opportunities. diff --git a/docs/data_management/transfer/globus/gcp_install.md b/docs/data_management/transfer/globus/gcp_install.md new file mode 100644 index 000000000..48f8ffc6c --- /dev/null +++ b/docs/data_management/transfer/globus/gcp_install.md @@ -0,0 +1,120 @@ +# How to Install Globus Connect Personal (GCP) + +[Globus Connect Personal](https://www.globus.org/globus-connect-personal) is software designed for installation on local machines (laptops, desktops, workstations, and other self-managed systems) that lets you create and use a personal Globus Collection (endpoint). It is available for macOS, Windows, and Linux, and Globus provides excellent installation documentation for each platform: + +- [MacOS](https://docs.globus.org/globus-connect-personal/install/mac/) +- [Windows](https://docs.globus.org/globus-connect-personal/install/windows/) +- [Linux](https://docs.globus.org/globus-connect-personal/install/linux/) + +We do not cover Linux installation here; please refer to the official documentation if you need to install GCP on a Linux system. Below are the GCP installation instructions for Windows and MacOS: + +- [Windows](#installing-gcp-on-windows) +- [MacOS](#installing-gcp-on-macos) + +## Installing GCP on Windows + +The following steps will guide you to install Globus Connect Personal (GCP) on your computer running Windows. + +1. Navigate to the [Globus Connect Personal official page](https://www.globus.org/globus-connect-personal) and scroll down to find the GCP Windows version. Click on the "INSTALL NOW" button in the red box, as shown in the image below, to be taken to the official installation instructions and download link for GCP for Windows. The "INSTALL NOW" button is located below the text "Globus Connect Personal for Windows". + + ![Globus Connect Personal official page showing operating system options with links. Windows is in the center of the three.](../images/go-gcp-install/win/001-select.png) + +1. Click the "Download Globus Connect Personal" link on the instructions page, as shown below. This will redirect you to the GCP for Windows installer. + + ![Globus Connect Personal for Windows official instructions page with link to download GCP for Windows.](../images/go-gcp-install/win/002-instructions.png) + +1. As shown in the image below, click the "Download Globus Connect Personal for Windows" button to download the installer to download the installer to your computer. + + ![Download GCP Installer for Windows](../images/go-gcp-install/win/003-download.png) + +1. Find the installer on your computer and open it. Select where you would prefer to have your GCP installed and click the "Install" button. + + ![GCP installer window showing selected path and install button.](../images/go-gcp-install/win/004-select-path.png) + + Please note you must have administrator permissions, to do this. If you are unable to do so, you will need to contact the IT department responsible for managing your computer. + +1. When the installation is complete, click the "Finish" button to complete the GCP installation. + + ![Completing GCP Setup highlighting Finish button.](../images/go-gcp-install/win/005-finish.png) + +1. Following installation, GCP will launch in a new window. If it does not, look for it in your Start Menu. + + When GCP has started, click the "Log In" button to authenticate with Globus to begin the Collection setup process. This is a one-time setup to configure GCP to allow your machine to act as a Collection, enabling research data transfer with your computer. + + ![GCP Setup window with Log In button.](../images/go-gcp-install/win/006-authenticate.png) + + Note that if you uninstall and reinstall GCP, you will need to complete this process again. You should not need to repeat this process otherwise. + +1. Grant the required consents. This is required to set up your computer as a Collection. Also provide a name for your consents. We recommend choosing a name that is short, memorable, and related to the purpose for the Collection. + + ![GCP Setup form with required consents and Collection name field.](../images/go-gcp-install/common/consents.png) + +1. Enter the details for your GCP Collection, and click save to continue. The following list describes the fields in the form shown below. + + - **Owner Identity:** is the person responsible for this Collection. This field should already be filled with UAB Campus or XIAS email address. If not, please select that email address here. + - **Collection Name:** is the name for the Collection. This should be filled with the name of the Collection from the previous step. + - **Description:** Feel free to enter descriptive information about the Collection here. This information will be displayed in the Globus Web App when the Collection is viewed by others. + - **High Assurance:** Only check this box if the Collection has or will have PHI, HIPAA, or other protected data. If this is the case, please ensure that you have already completed a risk assessment with UAB Enterprise IT. + + ![GCP Setup Collection details](../images/go-gcp-install/common/collection-details.png) + +1. GCP Setup is now complete on your computer. Your computer is now serving your new Globus Collection and may be used to transfer data. Click "Exit Setup" to close the window. + +1. After installation, locate your Windows System Tray. In it you should see a white lowercase letter "g" in a filled blue circle. If you do not, try finding the Globus Connect Personal application in your start menu and starting the application. + + ![GCP Icon in Windows system tray.](../images/go-gcp-install/win/009-system-tray-icon.png) + +By default your Documents folder (typically `C:/Users/%username%/Documents`) is listed in your new Globus Collection. To change it continue on with [How Do I Choose Specific Folders Using Globus Connect Personal?](../globus/gcp_setup.md#how-do-i-choose-specific-folders-to-share-using-globus-connect-personal). + +### Installing GCP on MacOS + +The following steps will guide you to install Globus Connect Personal (GCP) on your computer running MacOS. + +1. Navigate to the [Globus Connect Personal official page](https://www.globus.org/globus-connect-personal) and scroll down to find the GCP MacOS version. Click on the "INSTALL NOW" button in the red box in the image below to be taken to the official installation instructions and download link for GCP for MacOS. The "INSTALL NOW" button is located below the text "Globus Connect Personal for Mac". + + ![Globus Connect Personal official page showing operating system options with links. MacOS is the left of the three.](../images/go-gcp-install/mac/001-select.png) + +1. Click the "Download Globus Connect Personal" link on the instructions page, as shown below. This will redirect you to the GCP for MacOS installer. + + ![Globus Connect Personal for MacOS official instructions page with link to download GCP for MacOS.](../images/go-gcp-install/mac/002-instructions.png) + +1. As shown in the image below, click the "Download Globus Connect Personal for MacOS" button to download the installer to download the installer to your computer. + + ![Download GCP Installer for Windows](../images/go-gcp-install/mac/003-download.png) + +1. Find the installer on your computer and open it. A new window will pop-up asking you to drag the Globus Connect Personal app into the Application folder on your computer. Do so to install GCP. + + ![Copy Mac GCP App into the Applications Folder](../images/go-gcp-install/mac/004-install.png) + +1. When the above step is completed navigate to your Application folder and look for the "Globus Connect Personal" application. Open it to proceed. + + ![GCP application in the Applications Folder](../images/go-gcp-install/mac/005-application.png) + +1. When GCP has started, click on "Log In" to authenticate with Globus to begin the Collection setup process. This is a one-time setup to configure GCP to allow your machine to act as a Collection, enabling research data transfer with your computer. + + ![GCP Setup window with Log In button.](../images/go-gcp-install/mac/006-authenticate.png) + + Note that if you uninstall and reinstall GCP, you will need to complete this process again. You should not need to repeat this process otherwise. + +1. Grant the required consents. This is required to set up your computer as a Collection. Also provide a name for your Collection. We recommend choosing a name that is short, memorable, and related to the purpose for the Collection. + + ![GCP Setup form with required consents and Collection name field.](../images/go-gcp-install/common/consents.png) + +1. Enter the details for your GCP Collection, and click save to continue. The following list describes the fields in the form shown below. + + - **Owner Identity:** is the person responsible for this Collection. This field should already be filled with UAB Campus or XIAS email address. If not, please select that email address here. + - **Collection Name:** is the name for the Collection. This should be filled with the name of the Collection from the previous step. + - **Description:** Feel free to enter descriptive information about the Collection here. This information will be displayed in the Globus Web App when the Collection is viewed by others. + - **High Assurance:** Only check this box if the Collection has or will have PHI, HIPAA, or other protected data. If this is the case, please ensure that you have already completed a risk assessment with UAB Enterprise IT. + + ![GCP Setup Collection details](../images/go-gcp-install/common/collection-details.png) + +1. GCP Setup is now complete on your computer. Your computer is now serving your new Globus Collection and may be used to transfer data. Click "Exit Setup" to close the window. + +1. After installation locate your MacOS notification, generally at the top-right of your display. In it you should see a white lowercase letter "g" in a fille black circle. + + ![GCP Icon in MacOS notification area.](../images/go-gcp-install/mac/009-notification-area-icon.png) + +By default the `/Users/` folder is listed in your new Globus Collection. To change it continue on with [How Do I Share Specific Folders Using Globus Connect Personal?](../globus/gcp_setup.md#how-do-i-choose-specific-folders-to-share-using-globus-connect-personal). + +If you have any questions or run into any issues, feel free to [Contact Us](../../../help/support.md#how-to-request-support). diff --git a/docs/data_management/transfer/globus/gcp_setup.md b/docs/data_management/transfer/globus/gcp_setup.md new file mode 100644 index 000000000..0570d3c24 --- /dev/null +++ b/docs/data_management/transfer/globus/gcp_setup.md @@ -0,0 +1,81 @@ +# How to Use Globus Connect Personal (GCP) + +Globus Connect Personal (GCP) is a tool that lets you turn your local computer into a Globus endpoint so you can transfer data to and from the Globus ecosystem. + +To set up a local (personal) endpoint and use GCP, you must first install it by following the instructions on our [installation GCP](./gcp_install.md) page. + +## How Do I Choose Specific Folders to Share Using Globus Connect Personal? + +### Choose Specific Folders on Windows + +1. In your Windows system tray, locate the icon that looks like a small letter "g" in a circle. This is the icon for Globus Connect Personal. If you cannot locate the icon in the system tray, then open the Globus Connect Personal app on your computer and look for it again. + + ![Expanded system tray showing icon of a small letter "g" in a circle.](../images/go-choose-folder/win/001-sys-tray.png) + +1. Right-click the icon to open the context menu and click "Options..." + ![Context menu of Globus system tray icon showing options.](../images/go-choose-folder/win/002-context-menu.png) + +1. A new window will appear with a tab labelled Access. In the Access tab is an interface to configure folders available on your GCP Collection. For most use cases, you should not check the writeable checkbox. Below is a summary of what each part of the menu does. + + - **(1) Accessible Folders** table with Folder, Shareable and Writeable columns. Any folder listed here will appear on your GCP Collection. Your research data folder or directories must be listed here to be shareable. + - **(2) Shareable** column checkboxes controlling which folders can be shared with other users. Each of your research data directories must have this checkbox ticked to be shareable from the Collection. **Check this box only if you want to share your data with others.** + - **(3) Writeable** column checkboxes controlling which folders can be written to by other users. If a folder is shared with other users, then they will be able to add, delete, or change the contents. We recommend against ticking these boxes for Research Cores serving data to customers. **Check this box only if you want others to be able to change your data.** + - **(4) Plus `+` and minus `-` buttons** that allow you to add or remove folders from the list. + - **(5) Save** button which saves changes made to this tab of the options. + + ![Access tab of GCP options menu showing the default settings.](../images/go-choose-folder/win/003-access-tab-default.png) + +1. Use the plus `+` and minus `-` buttons to add your research data folders and remove other folders, as needed. Click the "Shareable" checkbox next to each research data folder. Click "Save" when finished. + + In this example, we removed the default `C:/Users/%username%/Documents` folder with the minus `-` button and added the `D:/data` folder with the `+` button and check the "Shareable" box. You will want to pick the folder where your research data is stored. + + ![Access tab of GCP options menu showing new settings.](../images/go-choose-folder/win/004-access-tab-changed.png) + +1. Click the "General" tab. The "General" tab enables you to control some settings for the application itself and which folder is the default folder. The default folder will be the first one shown when accessing the Collection. + + - **(1) Run when Windows starts** checkbox enabling starting Globus Connect Personal when you start Windows. **Check this box if GCP should always be on when the computer is on.** + - **(2) Home Folder** text field that lets you choose which folder will be the default folder for your Collection. We recommend setting this to your primary shared folder from the previous step to simplify navigating your Collection in the Globus Web App. + - **(3) Save** button which saves changes made to this tab of the options. Be sure to click "Save" if you make changes here. + + ![General tab of GCP options menu showing default settings.](../images/go-choose-folder/win/005-general-tab-default.png) + +1. Check "Run when Windows starts" if needed. Change the "Home Folder" to match your research data folder. Click "Save" when done. + + In this example, we set the "Home Folder" to match the research data folder, `D:/data` we added in a previous step. If you have multiple research directories to share, you will need to choose just one for this field. Be sure to click save when you are done. + + ![General tab of GCP options menu](../images/go-choose-folder/win/006-general-tab-changed.png) + +To verify the existence and accessibility of your Collection proceed to [How Do I Find Collections I Created or Own?](../globus/globus_organization_tutorial.md#how-do-i-find-collections-i-created-or-own) + +### Choose Specific Folders on MacOS + +1. In your MacOS notification area, locate the icon that looks like a small letter "g" in a circle. This is the icon for Globus Connect Personal. If you cannot locate the icon in the notification area, then open the Globus Connect Personal app on your computer and look for it again. + + ![Notification area showing icon of a small letter "g" in a circle.](../images/go-choose-folder/mac/001-notification-area.png) + +1. Right-click or command-click the icon to open the context menu. Click "Preferences…​". + + ![Context menu of Globus system tray icon showing preferences.](../images/go-choose-folder/mac/002-context-menu.png) + +1. A new window will appear with a tab labelled "Access". Click the "Access" tab if it is not already selected. In this "Access" tab is an interface to configure folders available on your GCP Collection. For most use cases, you should not check the writeable checkbox. Below is a summary of what each part of the menu does. + + - **(1) Accessible Directories and Files** table with "Directory or File", Shareable and Writeable columns. Any folder listed here will appear on your GCP Collection. Your research data folder or directories must be listed here to be shareable. + + + !!! note + + The terms Directories and Folders are synonyms here. + + + + - **(2) Shareable** column checkboxes controlling which folders can be shared with other users. Each of your research data directories must have this checkbox ticked to be shareable. **Check this box only if you want to share your data with others.** + - **(3) Writeable** column checkboxes controlling which folders can be written to by other users. If a folder is shared with other users, then they will be able to add, delete, or change the contents. We recommend against ticking these boxes for Research Cores serving data to customers. **Check this box only if you want others to be able to change your data.** + - **(4) Plus `+` and minus `-`** buttons that allow you to add or remove folders from the list. + + ![Access tab of GCP options menu showing the default settings.](../images/go-choose-folder/mac/003-access-tab.png) + +1. Use the plus `+` and minus `-` buttons to add your research data folders and remove other folders, as needed. Click the "Shareable" checkbox next to each research data folder. Click "Save" when finished. + +To verify the existence of your collection and to find collections you created or own, please refer to [How Do I Find Collections I Created or Own?](../globus/globus_organization_tutorial.md#how-do-i-find-collections-i-created-or-own). + +If you have any questions, please [Contact US](../../../help/support.md#how-to-request-support). diff --git a/docs/data_management/transfer/tutorial/globus_group_management.md b/docs/data_management/transfer/globus/globus_group_management.md similarity index 73% rename from docs/data_management/transfer/tutorial/globus_group_management.md rename to docs/data_management/transfer/globus/globus_group_management.md index 0c5744062..e5ae6648b 100644 --- a/docs/data_management/transfer/tutorial/globus_group_management.md +++ b/docs/data_management/transfer/globus/globus_group_management.md @@ -2,7 +2,7 @@ Organizations can manage students, staff, collaborators, and customers in Globus using the Group feature. Representatives can create groups and subgroups to manage and document associations between related people. With a Group, you can grant access to a Guest Collection to many people at once. -If you are new to Globus, we recommend starting with our [Globus Tutorials for Individual Researchers](./globus_individual_tutorial.md) first to familiarize yourself with how Globus is used. +If you are new to Globus, we recommend starting with our [Getting Started](./login_to_globus.md) and [Globus for Individual Researchers](../globus/globus_individual_tutorial.md) first to familiarize yourself with how Globus is used. Below is a list of topics covered in this article. @@ -33,85 +33,85 @@ Below is a list of topics covered in this article. ## Where Can I Find the Groups Page in the Globus Web App? -1. [Get onto the Globus Web App](./globus_individual_tutorial.md#how-do-i-get-onto-the-globus-web-app). +1. [Get onto the Globus Web App](../globus/login_to_globus.md#how-do-i-get-onto-the-globus-web-app). 1. In the left-hand navigation panel, click Groups to be taken to the Groups page. All Group management activities in this tutorial document start on this page, so take some time to familiarize yourself with getting to this point. - ![Groups page of the Globus Web App.](./images/common/groups.png) + ![Groups page of the Globus Web App.](../images/common/groups.png) ## How Do I Find a Group? 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). 1. If you are looking for new groups, ensure the "MY GROUPS" Checkbox is unchecked. If you are looking for groups you are already a member of, ensure the "MY GROUPS" Checkbox is checked. - ![My groups checkbox unchecked.](./images/common/groups-checked.png) + ![My groups checkbox unchecked.](../images/common/groups-checked.png) 1. Enter the name of the group you wish to find in the "Filter groups" search bar. - ![Entering uab-rclab in the search bar.](./images/gg-find-group/003-search-bar.png) + ![Entering uab-rclab in the search bar.](../images/gg-find-group/003-search-bar.png) 1. If the group you typed exists, you should see it appear in the results. If it does not, ensure that the "MY GROUPS" Checkbox is unchecked, if needed. - ![Results of search showing uab-rclab group.](./images/gg-find-group/004-results.png) + ![Results of search showing uab-rclab group.](../images/gg-find-group/004-results.png) 1. To view the newly-created Group's details page click its name in the results. If you are a Lab PI or Core Manager, please familiarize yourself with finding this details page. The details page is central to managing Groups. - ![Group page.](./images/gg-find-group/005-group-details-page.png) + ![Group page.](../images/gg-find-group/005-group-details-page.png) ## How Do I Request to Join a Group? 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). -1. [Find the Group you wish to join](./globus_group_management.md#how-do-i-find-a-group). Be sure the "MY GROUPS" Checkbox is unchecked or you will not be able to see new Groups in the search results. +1. [Find the Group you wish to join](../globus/globus_group_management.md#how-do-i-find-a-group). Be sure the "MY GROUPS" Checkbox is unchecked or you will not be able to see new Groups in the search results. - ![My Groups checkbox unchecked](./images/common/groups-unchecked.png) + ![My Groups checkbox unchecked](../images/common/groups-unchecked.png) 1. Click the "Join this Group" button. You can find this button on the main Groups page after searching for the Group. Note that some Groups may not allow invitation requests. To join restricted-access groups you will need to communicate with the Group's administrators. - ![Join this Group button on the Groups page.](./images/gg-join-group/001-general-page-button.png) + ![Join this Group button on the Groups page.](../images/gg-join-group/001-general-page-button.png) You can also find the button on the Group's details page. - ![Join this Group button on the Group details page.](./images/gg-join-group/001-specific-page-button.png) + ![Join this Group button on the Group details page.](../images/gg-join-group/001-specific-page-button.png) 1. Fill in the form and click the "Submit Application" button. Other groups may have additional fields to fill out. - ![Example form to join group.](./images/gg-join-group/002-form.png) + ![Example form to join group.](../images/gg-join-group/002-form.png) 1. You will see a notice of your pending membership application and will need to wait for approval from the group administrators. - ![Pending membership notification.](./images/gg-join-group/003-pending.png) + ![Pending membership notification.](../images/gg-join-group/003-pending.png) ## How Do I Find and Accept an Invitation From a Group? 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). 1. Be sure the "MY GROUPS" Checkbox is checked. When checked, you should see any pending invitations in the list of groups. - ![Pending invitation for group in the Groups page with the my groups checkbox checked.](./images/gg-accept-group/001-pending.png) + ![Pending invitation for group in the Groups page with the my groups checkbox checked.](../images/gg-accept-group/001-pending.png) 1. Click the name of the group with a pending invitation, or the right arrow at the right side of the table entry. You will be taken to the Group's details page. Click "Accept Invitation" to join the group. Click "Decline Invitation" to dismiss the invitation without joining. - ![Group specific page with Accept Invitiation and Decline Invitation buttons.](./images/gg-accept-group/002-buttons.png) + ![Group specific page with Accept Invitiation and Decline Invitation buttons.](../images/gg-accept-group/002-buttons.png) 1. If you clicked the "Accept Invitation" button, you will be taken to a form. Fill out the form and click "Accept Invitation" to join the group. Other groups may have additional fields to fill out. - ![Accept invitation form.](./images/gg-accept-group/003-form.png) + ![Accept invitation form.](../images/gg-accept-group/003-form.png) ## How Do I Leave a Group? 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). -1. [Find the Group you wish to leave](./globus_group_management.md#how-do-i-find-a-group). It may help to check the "MY GROUPS" Checkbox to only show Groups you are a member of. +1. [Find the Group you wish to leave](../globus/globus_group_management.md#how-do-i-find-a-group). It may help to check the "MY GROUPS" Checkbox to only show Groups you are a member of. - ![My Groups checkbox checked](./images/common/groups-checked.png) + ![My Groups checkbox checked](../images/common/groups-checked.png) 1. Click the "Leave Group" button. Note that this action cannot be undone, but you may [Rejoin the Group](#how-do-i-request-to-join-a-group) or be [Invited to Rejoin](#how-do-i-find-and-accept-an-invitation-from-a-group). - ![Leave group button on the groups page.](./images/gg-leave-group/001-leave-button.png) + ![Leave group button on the groups page.](../images/gg-leave-group/001-leave-button.png) ## How Do I Create a Group? 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). 1. Click the "Create New Group" button at the top-right of the window. - ![Create new Group button on Groups page](./images/gg-create-group/001-create-group-button.png) + ![Create new Group button on Groups page](../images/gg-create-group/001-create-group-button.png) 1. Fill in the form. More information about each field is below. When you have finished, click the "Create Group" button to create the group. @@ -126,38 +126,38 @@ Below is a list of topics covered in this article. - **(7) Group Visibility** radio buttons: Click "Members of this Group" to hide the group from non-members. - **(8) Membership Visibility** radio buttons: Click "Administrators and Managers Only" to hide member names from each other. Core adminstrators should consider using the "Administrators and Managers Only" option for their top-level groups because it improves the user experience when members are looking for administrator names. - ![Create Group form.](./images/gg-create-group/002-create-group-form.png) + ![Create Group form.](../images/gg-create-group/002-create-group-form.png) 1. Once the Group is created you should be taken to the newly-created Group's details page. Veryify the settings you picked are correct by clicking the "Additional Properties" drop down label to expand it, as highlighted below. - ![Group details page on the Overview tab showing additional properties.](./images/gg-create-group/003-group-details-page.png) + ![Group details page on the Overview tab showing additional properties.](../images/gg-create-group/003-group-details-page.png) ## How Do I Create a Subgroup? Subgroups are also full-fledged groups that happen to have a parent-child relationship with another group. Specifically, subgroups have a parent, and the subgroup is a child of the parent group. As a consequence, all members of the child subgroup are also members of the parent group. Subgroups may also have subgroups of their own. Otherwise, subgroups behave like any other group. 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). -1. [Find the Group](./globus_group_management.md#how-do-i-find-a-group). It may help to check the "MY GROUPS" Checkbox to only show Groups you are a member of. +1. [Find the Group](../globus/globus_group_management.md#how-do-i-find-a-group). It may help to check the "MY GROUPS" Checkbox to only show Groups you are a member of. - ![My Groups checkbox checked](./images/common/groups-checked.png) + ![My Groups checkbox checked](../images/common/groups-checked.png) 1. On the Group's details page, click the "Subgroups" tab. - ![Group Overview tab with add a subgroup button.](./images/gg-create-subgroup/001-subgroup-tab.png) + ![Group Overview tab with add a subgroup button.](../images/gg-create-subgroup/001-subgroup-tab.png) 1. On the "Subgroups" tab, click the "Add a Subgroup" button. - ![Group Overview tab with add a subgroup button.](./images/gg-create-subgroup/002-create-subgroup.png) + ![Group Overview tab with add a subgroup button.](../images/gg-create-subgroup/002-create-subgroup.png) 1. You should be taken to the Group Creation form. Proceed from here as though you are [Creating a Group](#how-do-i-create-a-group). ## How Do I See Subgroups of a Group? 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). -1. [Find the Group](./globus_group_management.md#how-do-i-find-a-group). +1. [Find the Group](../globus/globus_group_management.md#how-do-i-find-a-group). 1. On the Group's details page, click the "Subroups" tab. Subgroups will be listed here. In this case, the only subgroup is "uab-rc-facilitation". - ![Subgroup tab of group details page.](./images/gg-view-subgroups/001-subgroup-tab-page.png) + ![Subgroup tab of group details page.](../images/gg-view-subgroups/001-subgroup-tab-page.png) ## How Do I Modify a Group or Subgroup? @@ -165,33 +165,33 @@ Subgroups are also full-fledged groups that happen to have a parent-child relati 1. [Find the Group You Want to Modify](#how-do-i-find-a-group). 1. Click the Settings tab and modify the settings as necessary. To review what the fields are used for, see [How Do I Create a Group?](#how-do-i-create-a-group) - ![Settings tab of Group details page](./images/gg-modify-group/001-settings-tab.png) + ![Settings tab of Group details page](../images/gg-modify-group/001-settings-tab.png) ## How Do I Make a Group Invite Only? 1. [Prepare to modify the Group](#how-do-i-modify-a-group-or-subgroup). 1. Under "Policies" find the "Membership Visibility" controls and select "Administrators and Managers Only" to hide Group members from each other. This can be useful for groups with many members to make it easier to find managers and administrators. - ![Group settings page with group visibility controls highlighted.](./images/gg-modify-group-invite-only/001-request-to-join-checkbox.png) + ![Group settings page with group visibility controls highlighted.](../images/gg-modify-group-invite-only/001-request-to-join-checkbox.png) ## How Do I Keep Group Members From Seeing Each Other? 1. [Prepare to modify the Group](#how-do-i-modify-a-group-or-subgroup). 1. Under "Policies" find the "Membership Visibility" controls and select "Administrators and Managers Only" to hide Group members from each other. This can be useful for groups with many members to make it easier to find managers and administrators. - ![Group settings page with group visibility controls highlighted.](./images/gg-modify-group-hide-members/001-hide-members-radio-button.png) + ![Group settings page with group visibility controls highlighted.](../images/gg-modify-group-hide-members/001-hide-members-radio-button.png) ## How Do I Hide a Group From Others? 1. [Prepare to modify the Group](#how-do-i-modify-a-group-or-subgroup). 1. Under "Policies" find the "Group Visibility" controls and select "Members of the Group" to hide your group from non-members. This can be useful for groups with restricted or controlled access. - ![Group settings page with group visibility controls highlighted.](./images/gg-modify-group-hide-group/001-hide-group-radio-button.png) + ![Group settings page with group visibility controls highlighted.](../images/gg-modify-group-hide-group/001-hide-group-radio-button.png) ## How Do I Delete a Group or Subgroup? 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). -1. [Find the Group you wish to delete](./globus_group_management.md#how-do-i-find-a-group). It may help to check the "MY GROUPS" Checkbox. +1. [Find the Group you wish to delete](../globus/globus_group_management.md#how-do-i-find-a-group). It may help to check the "MY GROUPS" Checkbox. 1. Click the "Delete Group" button to delete the group. This action is highly destructive and cannot be undone. Be absolutely certain you know what you are doing before deleting a group. Subgroups will no longer have this group as their parent and cannot be added as subgroups of other groups. The parent-child relationship between this group and its subgroups will be permanently destroyed. Subgroups will become independent groups following deletion and will not be deleted. @@ -202,35 +202,35 @@ Subgroups are also full-fledged groups that happen to have a parent-child relati Deleting a group is a highly destructive action. Deleting a group cannot be undone. Be absolutely certain you know what you are doing before deleting a group. - ![Delete group button.](./images/gg-delete-group/001-delete-button.png) + ![Delete group button.](../images/gg-delete-group/001-delete-button.png) ## How Do I Invite Others to a Group? 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). -1. [Find the Group you wish to invite others to join](./globus_group_management.md#how-do-i-find-a-group). It may help to check the "MY GROUPS" Checkbox. +1. [Find the Group you wish to invite others to join](../globus/globus_group_management.md#how-do-i-find-a-group). It may help to check the "MY GROUPS" Checkbox. 1. Click the "Invite Others" button to invite others to your group. - ![Invite others button on group specific page.](./images/gg-invite-to-group/001-invite-button.png) + ![Invite others button on group specific page.](../images/gg-invite-to-group/001-invite-button.png) 1. Type the name, email, or BlazerID of the person you wish to add into the search bar and click "Add". - ![Search bar and add button to find a user.](./images/gg-invite-to-group/002-search-and-add.png) + ![Search bar and add button to find a user.](../images/gg-invite-to-group/002-search-and-add.png) 1. Select the role to give the user when adding them. See [What Does Each Role Mean?](#what-does-each-role-mean) to help decide. - ![Select role from listed roles.](./images/gg-invite-to-group/003-select-role.png) + ![Select role from listed roles.](../images/gg-invite-to-group/003-select-role.png) 1. Click "Send Invitation" to invite the user. You should see the following confirmation page. They will need to follow instructions at [How Do I Find and Accept an Invitation From a Group?](#how-do-i-find-and-accept-an-invitation-from-a-group) to join. - ![Invitation confirmation page.](./images/gg-invite-to-group/004-confirmation.png) + ![Invitation confirmation page.](../images/gg-invite-to-group/004-confirmation.png) ## How Do I See Group Members? 1. [Get to the Groups page of the Globus Web App](#where-can-i-find-the-groups-page-in-the-globus-web-app). -1. [Find the Group](./globus_group_management.md#how-do-i-find-a-group). +1. [Find the Group](../globus/globus_group_management.md#how-do-i-find-a-group). 1. On the Group's details page, click the "Members" tab. Members will be listed here. - ![Members tab of group details page.](./images/gg-view-members/001-members-tab-page.png) + ![Members tab of group details page.](../images/gg-view-members/001-members-tab-page.png) ## How Do I Modify Group Members? @@ -238,36 +238,36 @@ Subgroups are also full-fledged groups that happen to have a parent-child relati 1. [Find the Group you wish to change](#how-do-i-find-a-group). 1. On the Group's details page, click the Members tab to see a list of Group members. - ![Members tab of Group details page.](./images/gg-modify-members/001-members-tab.png) + ![Members tab of Group details page.](../images/gg-modify-members/001-members-tab.png) 1. Locate the member. Click the right arrow button at the right-hand side of the member's entry to be taken to their membership page. - ![Member's membership page.](./images/gg-modify-members/002-membership-page.png) + ![Member's membership page.](../images/gg-modify-members/002-membership-page.png) ## How Do I Change a Group Member's Role Level? 1. [Prepare to modify Group memberships](#how-do-i-modify-group-members). 1. Click the pencil icon just to the right of the member's role to make changes to the role as highlighted in the image. - ![Edit role icon in the member's membership page.](./images/gg-modify-members-role/003-pencil-edit.png) + ![Edit role icon in the member's membership page.](../images/gg-modify-members-role/003-pencil-edit.png) 1. After clicking the icon, new controls become available in place of the existing role. Select the new role from the drop-down menu, then click "Save" to make the change permanent. - ![Controls to modify member role.](./images/gg-modify-members-role/004-role-controls.png) + ![Controls to modify member role.](../images/gg-modify-members-role/004-role-controls.png) ## What Does Each Role Mean? Different roles have different purposes. The Administrator role is intended to be for managing both the group itself and its members. The Manager role is for managing members of the group. The Member role is purely for granting access to collections associated with the group. The table below contains more details. -{{ read_csv('data_management/transfer/tutorial/res/globus-group-roles.csv', keep_default_na=False, colalign=("left","center","center","center")) }} +{{ read_csv('data_management/transfer/globus/res/globus-group-roles.csv', keep_default_na=False, colalign=("left","center","center","center")) }} ## How Do I Remove a Group Member? 1. [Prepare to modify Group memberships](#how-do-i-modify-group-members). 1. Click the "Remove Membership" button to immediately remove the member from the group. There is no confirmation for this action. - ![Membership page with highlighted remove member button.](./images/gg-modify-members-remove/002-membership-page.png) + ![Membership page with highlighted remove member button.](../images/gg-modify-members-remove/002-membership-page.png) 1. The page should update to reflect the change immediately. Instead of the "Remove Membership" button, you should see "Add Membership" and "Send Invitation" buttons. - ![Membership page with highlighted remove member button.](./images/gg-modify-members-remove/003-membership-page-after.png) + ![Membership page with highlighted remove member button.](../images/gg-modify-members-remove/003-membership-page-after.png) diff --git a/docs/data_management/transfer/globus/globus_individual_tutorial.md b/docs/data_management/transfer/globus/globus_individual_tutorial.md new file mode 100644 index 000000000..513b8f5f8 --- /dev/null +++ b/docs/data_management/transfer/globus/globus_individual_tutorial.md @@ -0,0 +1,429 @@ +# Globus for Individual Researchers + +Do you have data and need to move it to a different computer system? Want to use a graphical interface to do it? Want a service that will attempt to resume interrupted transfers? Want enforced encryption for your transfers? Then Globus is right for you. + +These guides are intended for individual researchers who need to move data from one location to another. If you represet a group, manage a Research Core or lab and have never used Globus before, you'll want to start on this page. Then you will want to proceed to [Globus for Research Groups](../globus/globus_organization_tutorial.md). + +The resource below will teach you how to effectively use Globus for managing and transferring research data. We will explore what Globus is, why you might use it, how it works, and we'll walk you through the essential steps to get started. You will learn how to set up your Globus account, access the Globus application, find Collections shared with you, and search for Collections by name. + +Topics covered: + +1. [Why Globus?](#why-globus) +1. [How Does Globus Work?](#how-does-globus-work) +1. [Prerequisites](#prerequisites) +1. [How Do I Get Onto the Globus Web App?](./login_to_globus.md#how-do-i-get-onto-the-globus-web-app) +1. [How Do I Search for Collections by Name?](#how-do-i-search-for-collections-by-name) +1. [How Do I Find UAB Storage Collections?](#how-do-i-find-uab-storage-collections) +1. [How Do I Find Collections Shared with Me?](#how-do-i-find-collections-shared-with-me) +1. [How Do I Transfer Data Between Collections?](#how-do-i-transfer-data-between-collections) + 1. [How Do I Transfer between a Collection and Cheaha?](#how-do-i-transfer-between-a-collection-and-cheaha) + 1. [How Do I Transfer between a Collection and LTS?](#how-do-i-transfer-between-a-collection-and-lts) + 1. [How Do I Transfer between LTS and Cheaha?](#how-do-i-transfer-between-lts-and-cheaha) +1. [How Do I Check Transfer Status?](#how-do-i-check-transfer-status) + +## Why Globus? + +Globus is a data transfer ecosystem that simplifies the process of transferring, sharing, and managing large datasets. It is used by research institutions around the world to move research data between different storage devices, computer systems, and institutions. Globus has many research data oriented features, making it ideal for many research data transfer scenarios. Below is a list of features. + +- Straight-forward, browser-based, graphical interface. +- Compatible with [UAB Box](https://www.uab.edu/it/home/tech-solutions/file-storage/box). +- Compatible with [Long-Term Storage](../../storage/lts/index.md). +- Can be used with your laptop, desktop, or lab workstation via Globus Connect Personal (GCP). +- Transfers are automatically retried in the event of network or computer system outages. +- Transfers are encrypted end-to-end. Globus never sees your data. +- Suitable for transferring PHI and HIPAA data. Note: a UAB Enterprise IT risk assessment is required. + +## How Does Globus Work? + +Globus is an ecosystem of software intended to make research data transfer simpler. The Globus Web Application (Web App) at allows you to initiate transfers between any two Collections you have authorization to access. The Globus Connect Personal (GCP) software lets you turn any computer into a Globus Collection. At no point do Globus servers touch your research data. Instead, when you initiate a transfer between two Collections, the Globus application tells the two Collections that they need to talk to each other and data is sent directly between them. The Collections update the application with information you may need to know, such as how much data has transferred so far, how fast the transfer is proceeding, and any errors that occur. If the connection between Collections is interrupted for any reason, the Globus application will attempt to restart the transfer from where it left off. + +## Prerequisites + +For these guides, you will need your BlazerID or [XIAS ID](../../../account_management/xias/index.md) and password to authenticate using UAB Single Sign-On (SSO). + +## How Do I Search for Collections by Name? + +Please follow these instructions to search for a specific Collection by name. + +1. [Get onto the Globus Web App](./login_to_globus.md#how-do-i-get-onto-the-globus-web-app). +1. You should see the File Manager page. If not, click File Manager in the left-hand navigation panel. + + ![File Manager page.](../images/globus_002_nav_pane_file_manager.png) + +1. Click the Collection Search bar to open the Collection Search page. + + ![Collection search page](../images/common/collection-search-page.png) + +1. In the example below, we typed "uab box" to search for the UAB Box Collection, which connects to UAB's Box.com service. + + ![Globus Collection search page showing results of searching for "UAB Box".](../images/gi-search-collections/003-results.png) + +1. Click the name of the Collection to be taken back to the file manager page with the Collection filled in. + + ![File Manager page of the Globus Web App. One of the Collection Search bars is filled in with the previously selected Collection.](../images/gi-find-shared-collections/004-selected.png) + +If you can't find a particular Collection this way, but know it was shared with you, try [finding Collections shared with me](#how-do-i-find-collections-shared-with-me). Proceed on to learn [how to find UAB storage Collections](#how-do-i-find-uab-storage-collections). + +## How Do I Find UAB Storage Collections? + +UAB offers [multiple storage resources](../../index.md). The following resources may be accessed through Globus Collections. + +- Cheaha file system (GPFS) including [individual directories](../../storage/index.md#what-individual-storage-solutions-are-available) and [project directories](../../storage/cheaha_storage_gpfs/project_directories.md). +- [Long-Term Storage (LTS)](../../storage/lts/index.md). +- [UAB Box](https://www.uab.edu/it/home/tech-solutions/file-storage/box). + +To find these resources on Globus, use the following steps. + +1. [Get onto the Globus Web App](./login_to_globus.md#how-do-i-get-onto-the-globus-web-app). +1. You should see the File Manager page. If not click File Manager in the left-hand navigation panel. + + ![File Manager page.](../images/common/file-manager.png) + +1. Click the Collection Search bar to open the Collection Search page. + + ![Collection search page](../images/common/collection-search-page.png) + +1. In the search bar, type one of the following, depending on which resource you need, and select the appropriate entry. + + - **Cheaha Filesystem (GPFS)**: Type "UAB Cheaha" and select the "UAB RCS Cheaha HPC", the collection currently supported. + + ![UAB Box search results.](../images/gi-uab-collections/001-cheaha.png) + + - **Long-Term Storage (LTS)**: Type "UAB LTS" and select the entry labeled "UAB Research Computing LTS (Long Term Storage aka S3)". + + ![UAB Box search results.](../images/gi-uab-collections/002-lts.png) + + - **UAB Box**: Type "UAB Box" and select the entry labeled "UAB Box". + + ![UAB Box search results.](../images/gi-uab-collections/003-box.png) + +Proceed on to learn [how to find Collections shared with you](#how-do-i-find-uab-storage-collections). + +## How Do I Find Collections Shared With Me? + +Globus allows users to share Collections with others. Other researchers, labs, and Research Cores on campus may invite you to their Collections to share data with you. The following instructions let you view Collections shared with you. + +1. [Log in to Globus using your credentials](./login_to_globus.md#how-do-i-get-onto-the-globus-web-app) +1. You should see the File Manager page. If not click File Manager in the left-hand navigation panel. + + ![File Manager page.](../images/common/file-manager.png) + +1. Click the Collection Search bar to open the Collection Search page. + + ![Collection search page](../images/common/collection-search-page.png) + +1. Click the Shared With You tab. The list of Collections will be filtered down to all Collections others have granted you access to, which should help you find the Collections you need. + + ![Collection Search page of Globus Web App. Shared With You tab is selected and the list of Collections has been filtered down.](../images/gi-find-shared-collections/003-list.png) + +1. Click the name of the Collection to be taken back to the file manager page with the Collection filled in. + + ![File Manager page of the Globus Web App. One of the Collection Search bars is filled in with the previously selected Collection.](../images/gi-find-shared-collections/004-selected.png) + +If you can't find a particular Collection this way, but know its name, try [searching for Collections](#how-do-i-search-for-collections-by-name). + +## How Do I Transfer Data Between Collections? + +This section outlines common data transfer scenarios between Globus Collections and Research Computing storage systems. + +- [Between a Collection and Cheaha](#how-do-i-transfer-between-a-collection-and-cheaha) +- [Between a Collection and LTS](#how-do-i-transfer-between-a-collection-and-lts) +- [Between LTS and Cheaha](#how-do-i-transfer-between-lts-and-cheaha) + +### How Do I Transfer Between a Collection and Cheaha? + +Please use the following instructions to transfer data from a Collection to [Cheaha GPFS](../../storage/cheaha_storage_gpfs/index.md). These instructions may also be used for other buckets on LTS, provided you have access and know their file path. + +The direction of transfer may also be reversed to transfer data from Cheaha to a Collection. Note that some Collections may not allow you to transfer data back, such as some of those provided by Research Cores. + +These instructions can be generalized to any two Collections you have access to on Globus. + +1. [Log in to Globus using your credentials](./login_to_globus.md#how-do-i-get-onto-the-globus-web-app). + +1. Find the Collection [by filtering](#how-do-i-find-collections-shared-with-me) or [by searching](#how-do-i-search-for-collections-by-name) in the left side Collection Search bar. + +1. Once you have selected the Collection you wish to transfer data from, repeat the process to [search](#how-do-i-search-for-collections-by-name) for a Cheaha Collection in the Collection Search bar on the right side of the file manager page. Search for "UAB Cheaha" to find and select the "UAB RCS Cheaha HPC", the collection currently supported. + +1. When you select a Cheaha Collection, or any other High Assurance (HA) Collection or Guest Collection, you will be prompted to re-authenticate. Click the Continue button to do so, then select your UAB email address. + + ![High Assurance request for reauthentication.](../images/common/ha-authenticate.png) + + ![Identity selection showing a UAB email address.](../images/common/select-identity.png) + +1. At this point, your file manager page should look something like the following image. At this point, both Collection Search bars should have a Collection filled in. The left side should be the Collection you wish to transfer from. The right side should be a Cheaha Collection. You should see files and folders on both sides. + + ![File Manager page showing a Collection and Cheaha Collection.](../images/gi-transfer-collection-to-cheaha/005-ready-to-select.png) + +1. Locate the path on the source Collection side. Either type the path into the Path field manually, or use the graphical selection field to click on folder names to navigate the filesystem. + +1. Repeat the process on the Cheaha Collection side to locate the destination path on the Cheaha Collection side. + +1. Select the file and folders you wish to transfer on the source Collection side. Do so by clicking the checkboxes next to the file and folder names. + + ![File Manager page with a file selected in the left selection area. The start button is boxed in red.](../images/gi-transfer-collection-to-cheaha/008-selection.png) + +1. To start the transfer, click the "Start" button on the side you made your selections. A transfer will be started and you should see a green toast notification at the upper-right corner of the web page. Press the "X" button to dismiss the notification or click "View Details" to be taken to the Activity page to see more details about the transfer. + + ![File Manager page with a toast notification indicating the file transfer started successfully.](../images/gi-transfer-collection-to-cheaha/009-transfer-started.png) + +From here you can proceed to other related tutorials to initiate other transfers or +[return to the index](#globus-for-individual-researchers). + +- [How Do I check transfer status?](#how-do-i-check-transfer-status) +- [How Do I Transfer Between a Collection and LTS?](#how-do-i-transfer-between-a-collection-and-lts) +- [How Do I Transfer Between LTS and Cheaha?](#how-do-i-transfer-between-lts-and-cheaha) + +### How Do I Transfer Between a Collection and LTS? + + +!!! note + + These instructions assume that you have already an LTS allocation, and your LTS collection is already set up in Globus. If you don't yet have an LTS allocation and/or If this is your first time managing an LTS allocation with Globus, you will need to [Request LTS Allocations](../../storage/lts/index.md#requesting-an-allocation) and [add your LTS S3 access and secret keys to the LTS endpoint](#adding-lts-allocation-credentials). This is a one-time setup. + + +Please use the following instructions to transfer data from a Collection to a [bucket on LTS](../../storage/lts/index.md). These instructions may also be used for other directories on Cheaha, provided you have access and know their file path. + +The direction of transfer may also be reversed to transfer data from LTS to the destination Collection. Note that some Collections may not allow you to transfer data back, such as some of those provided by Research Cores. + +These instructions can be generalized to any two Collections you have access to on Globus. + +1. [Log in to Globus using your credentials](./login_to_globus.md#how-do-i-get-onto-the-globus-web-app). + +1. Find a Collection [by filtering](#how-do-i-find-collections-shared-with-me) or [by searching](#how-do-i-search-for-collections-by-name) in the left side Collection Search bar. + +1. Once you have selected the shared Collection you wish to transfer data from, repeat the process to [search](#how-do-i-search-for-collections-by-name) for the LTS Collection in the Collection Search bar on the right side of the file manager page. Search for "UAB LTS" to find it. + +1. When you select the LTS Collection, or an High Assurance (HA) Collection, you will be prompted to re-authenticate. Click the "Continue" button to do so, then select your UAB email address. + + ![High Assurance request for re-authentication.](../images/common/ha-authenticate.png) + + ![Identity selection showing a UAB email address.](../images/common/select-identity.png) + +1. At this point, your file manager page should look something like the following image. Both Collection Search bars should have a Collection filled in. The left side should be the Collection you wish to transfer from. The right side should be the "UAB Research Computing LTS (Long Term Storage aka S3)" Collection. You should see files and folders on both sides. + + ![File Manager page showing a Collection and LTS Collection.](../images/gi-transfer-collection-to-lts/005-ready-to-select.png) + +1. Locate the path on the source Collection side. Either type the path into the Path field manually, or use the graphical selection field to click on folder names to navigate the filesystem. + +1. Repeat the process on the LTS Collection side to locate the destination path on the LTS Collection side. + +1. Select the file and folders you wish to transfer on the source Collection side. Do so by clicking the checkboxes next to the file and folder names. + + ![File Manager page with a file selected in the left selection area. The start button is boxed in red.](../images/gi-transfer-collection-to-lts/008-selection.png) + +1. To start the transfer, click the "Start" button on the side you made your selections. A transfer will be started and you should see a green toast notification at the upper-right corner of the web page. Press the "X" button to dismiss the notification or click "View Details" to be taken to the Activity page to see more details about the transfer. + + ![File Manager page with a toast notification indicating the file transfer started successfully.](../images/gi-transfer-collection-to-lts/009-transfer-started.png) + +From here you can proceed to other related tutorials to initiate other transfers or +[return to the index](#globus-for-individual-researchers). + +- [How do I check transfer status?](#how-do-i-check-transfer-status) +- [How Do I Transfer Between a Collection and Cheaha?](#how-do-i-transfer-between-a-collection-and-cheaha) +- [How Do I Transfer Between LTS and Cheaha?](#how-do-i-transfer-between-lts-and-cheaha) + +### How Do I Transfer Between LTS and Cheaha? + +Please use the following instructions to transfer data from a [bucket on LTS](../../storage/lts/index.md) to a project directory on Cheaha. These instructions may also be used for other buckets on LTS and directories on Cheaha, provided you have access and know their file paths. + +The direction of transfer may also be reversed to transfer data from Cheaha to LTS. + +These instructions can be generalized to any two Collections or Collections you have access to on Globus. + +1. [Log in to Globus using your credentials](./login_to_globus.md#how-do-i-get-onto-the-globus-web-app). + +1. Find the LTS Collection by [searching](#how-do-i-search-for-collections-by-name) in the Collection Search bar on the left side of the file manager page. Search for "UAB LTS" to find it. + +1. Once you have selected the LTS Collection, repeat the process to [search](#how-do-i-search-for-collections-by-name) for a Cheaha Collection in the Collection Search bar on the right side of the file manager page. Search for "Cheaha cluster" to find it. + + Pay close attention in choosing which of the two you need. Choosing incorrectly could lead to slow transfers. Answer the following questions to help you decide. Is the first Collection you selected... + + - ...part of a lab or Research Core on campus? Select "on-campus". + - ...on a computer on the UAB Campus Network or UAB Wifi or the UAB Campus VPN? Select "on-campus". + - ...at a different institution? Select "off-campus". + - ...on a computer on a home network? Select "off-campus". + +1. Both the LTS and Cheaha Collections are High Assurance (HA) Collections and you will be prompted to re-authenticate. Click the "Continue" button to do so, then select your UAB email address. + + ![High Assurance request for reauthentication.](../images/common/ha-authenticate.png) + + ![Identity selection showing a UAB email address.](../images/common/select-identity.png) + +1. At this point, your file manager page should look something like the following image. Both Collection Search bars should have a Collection name filled in. The left side should be the LTS Collection. The right side should be a Cheaha Collection. You should see files and folders on both sides. + + ![File Manager page showing a shared Collection and LTS Collection.](../images/gi-transfer-lts-to-cheaha/005-ready-to-select.png) + +1. Locate the source path on the LTS Collection side. Either type the path into the Path field manually, or use the graphical selection field to click on folder names to navigate the filesystem. + +1. Repeat the process on the Cheaha Collection side to locate the destination path on the Cheaha Collection side. + +1. Select the file and folders you wish to transfer on the source Collection side. Do so by clicking the checkboxes next to the file and folder names. + + ![File Manager page with a file selected in the left selection area. The start button is boxed in red.](../images/gi-transfer-lts-to-cheaha/008-selection.png) + +1. To start the transfer, click the "Start" button on the side you made your selections. A transfer will be started and you should see a green toast notification at the upper-right corner of the web page. Press the "X" button to dismiss the notification or click "View Details" to be taken to the Activity page to see more details about the transfer. + + ![File Manager page with a toast notification indicating the file transfer started successfully.](../images/gi-transfer-lts-to-cheaha/009-transfer-started.png) + +From here you can proceed to other related tutorials to initiate other transfers or +[return to the index](#globus-for-individual-researchers). + +- [How do I check transfer status?](#how-do-i-check-transfer-status) +- [How Do I Transfer Between a Collection and Cheaha?](#how-do-i-transfer-between-a-collection-and-cheaha) +- [How Do I Transfer Between a Collection and LTS?](#how-do-i-transfer-between-a-collection-and-lts) + +## How Do I Check Transfer Status? + +To check the status of your transfers, please follow the instructions below. + +1. [Log in to Globus using your credentials](./login_to_globus.md#how-do-i-get-onto-the-globus-web-app). + +1. In the Globus Web App, click "Activity" in the left navigation panel to go to the Activity page. There will be a list overview of transfers, with the most recent at the top. + + ![Globus Web App Activity page showing a successful transfer and failed transfer.](../images/gi-check-transfer-status/001-activity.png) + +1. To see more details about a transfer, click the transfer title. There will be two tabs. The Overview tab will have information and statistics about the transfer. The Event Log tab will have information about events that occurred during transfer, including start, stop, and any errors. The Event Log is useful for diagnosing issues with failed transfers. + + ![Overview page for unsuccessful transfer.](../images/gi-check-transfer-status/002-status.png) + + +!!! note + + File permissions from the source will not be copied to the destination. Please read more at [this ask.ci FAQ](https://ask.cyberinfrastructure.org/t/why-cant-other-users-access-data-i-transferred-to-a-project-space-on-cheaha/2527/3). + + +## Transfer and Sync Options + +Between the two "Start" buttons on the "File Manager" page is a "Transfer & Timer Options" drop down menu. Click that button to change the options. More information on each option. A brief summary of the options under "Transfer Settings" are... + +![!Transfer and Timer Options pane showing multiple options.](../images/globus_040_transfer_and_sync_options.png) + +1. sync - Sync files only, rather than create new files. +1. delete files - Delete any files on the target that are not on the source. Useful for forcing identical filesystems when syncing. +1. preserve source - Copies file "modified time" metadata. +1. verify integrity - Verifies that checksums are identical on source and target after transfer completes. The default option is to verify integrity. Its highly recommended to leave this unchecked. +1. encrypt transfer - Encrypts data before leaving source and decrypts after arriving at destination. Recommended for all transfers, required and enforced for all UAB endpoints. It is checked by default. +1. skip files - Skips source files that cause errors during the transfer. Otherwise the entire transfer will stop when an error is encountered. +1. quota fail - Fails instead of retries when the target storage quota is exceeded. +1. Apply filter - Filter rules can be used to customize and fine-tune the transfer process, ensuring that only the desired files/directories are transferred while excluding others based on the rules defined. + +Below the "Transfer Settings", there are additional options available that you can enable to customize notification of success/failure, and schedule the transfer to occur at specific time. + +### Common Errors + +1. File Not Found - This may mean that a file was not readable by Globus. Check that the file hasn't moved or changed names during the transfer. It is recommended to not modify files while they are being transferred by Globus. +1. Permission Denied - Globus is not able to access the files because permissions do not allow it. For Globus Connect Personal, be sure the containing folder is on the "Accessible Folders" list. Be sure that your Cheaha account has access to read the file. + +### More Information + +A [Globus FAQ](https://docs.globus.org/faq/globus-connect-endpoints/) is available for additional information on endpoints and transfers. + +## Long-Term Storage S3 (LTS) Connector + + +!!! important + + [LTS](../../storage/lts/index.md) behaves differently from other file systems and comes with a few possible pitfalls. Keep in mind the following three rules: (1) all data must be in buckets, (2) buckets are only allowed in the root folder, and (3) buckets must have unique names. + +UAB Researcher Computing has subscriptions to connectors for LTS and other types of filesystems. To use the UAB [LTS](../../storage/lts/index.md) Connector, enter "UAB LTS" into the Collection search box. Select the endpoint labeled "UAB Research Computing LTS (Long Term Storage aka S3)". If you have stored data within LTS already you should see a list of folders, otherwise you will see an empty space where folders may be placed. Each folder corresponds to a [bucket](../../storage/lts/index.md#terminology) in LTS. To create a bucket, click "New Folder" in the "File Manager" window in Globus. Note that buckets must have globally unique names. Read on for more information about possible pitfalls. + + ![create bucket via globus](../images/globus-create-lts-bucket.png) + +### Adding LTS Allocation Credentials + +Before you can manage an LTS allocation using Globus, you'll need to add the [LTS S3 Access and Secret Keys](../../storage/lts/index.md#requesting-an-allocation) to the LTS endpoint. To do so, please follow the instructions given below. + +1. In your browser, navigate to and login using UAB SSO. +1. Click "Collections" in the left-hand navigation pane. +1. In the search field at the top of the "Collection" page, enter "UAB LTS" and then click the search button. Be sure to uncheck "Recently Used" if it is checked. You should see a collection titled "UAB Research Computing LTS (Long Term Storage aka S3)" appear in the search results. +1. Click the right arrow indicator at the right-hand side of the "UAB Research Computing LTS" result. You should be taken to the Overview page for the "UAB Research Computing LTS" endpoint. + + ![!Image showing Globus collections page with UAB LTS in search box, UAB Research Computing LTS endpoint visible in results, and right arrow highlighted.](../images/globus-lts-s3-credentials-01.png) + +1. Near the top of the Overview page, click the "Credentials" tab". After some time, you should see a form requesting your "AWS IAM Access Key ID" and "AWS IAM Secret Key". +1. Enter the two requested keys in the appropriate boxes, then press "Continue". For data security reasons, you may be asked to authenticate again with your UAB SSO credentials. If you have more than one set of keys, you will need to choose which to enter. At this time, there doesn't appear to be a way to enter or use more than one set of credentials at a time with Globus. + + ![!Image showing Register a Credential form on the UAB Research Computing LTS endpoint Overview page showing artificial keys entered into the appropriate text fields.](../images/globus-lts-s3-credentials-02.png) + +1. You should be taken back to the UAB Research Computing LTS endpoint Overview page with the Credentials tab selected. There should now be one entry on this page showing your AWS IAM Access Key ID and your Globus Identity (i.e., your `BlazerID@uab.edu` email address). If you have multiple Access Keys and ever wish to change which one is being used with Globus, then click the "Trash Can" icon next to the entry on this page to delete it, and start the key entry process over from the previous step. + + ![!Image showing UAB Research Computing LTS endpoint Overview page showing Access Credentials entry with AWS IAM Access Key ID and Globus Identity.](../images/globus-lts-s3-credentials-03.png) + +At this point you are able to access the LTS allocation associated with the Access Key you entered, and any buckets which have granted access to that Access Key. If you have more than one Access Key, e.g. for each of your personal and project allocations, you will need to choose which key to enter above. If you ever wish to change credentials, then click the "Trash Can" icon next to the entry shown in step 6, above, and start the key entry process over. + +### Data Must Be in Buckets + +All data transferred to LTS must be placed in a bucket, and may _not_ be placed directly into the root directory. Attempting to move data to the root directory will result in an unhelpful error message in the "Activity" window. + +![!unhelpful error message for data placed in the root LTS directory](../images/globus_lts_no_bucket_error_001.png) + +Clicking on the "view event log" link shows the following. + +![!unhelpful event log message](../images/globus_lts_no_bucket_error_002.png) + +```text +Error (transfer) +Endpoint: UAB Research Computing LTS (Long Term Storage aka S3) (184408b4-d04b-4513-9912-8feeb6adcab3) +Server: m-a201b5.9ad93.a567.data.globus.org:443 +Command: STOR /test.log +Message: The connection to the server was broken --- +Details: an end-of-file was reached\nglobus_xio: An end of file occurred\n +``` + +### Buckets Must Have Globally Unique Names + +When creating new buckets, the name must be unique across all buckets on the system. If a duplicate bucket name, for example `first-test-bucket`, is entered, a long error message will appear in a small space next to the new bucket name. For readability, the expanded message is shown below. + +![!large error message in small space](../images/globus_lts_duplicate_name_error_001.png) + +```text +Remote Endpoint Failure: Path already exists, Error (mkdir) +Endpoint: UAB Research Computing LTS (Long Term Storage aka S3) (184408b4-d04b-4513-9912-8feeb6adcab3) +Server: 138.26.220.68:443 +Message: Path '/first-test-bucket/' already exists +``` + +At first glance, requiring unique names across all buckets on the system may sound very restrictive, but it is necessary for LTS to be as fast as it is. Fortunately, there is an easy way to deal with the limitation. See our LTS section on [good naming practice](../../storage/lts/index.md#avoiding-duplicate-names-for-buckets) for how to avoid duplicate names. For example, if you want to name a bucket `ai-lab` for storing data related to the entire AI lab or a specific dataset, you can append a universally unique identifier (UUID) to the name. To generate a UUID, visit , and a 16-byte UUID will be automatically generated. You can then copy it and append it to the name `ai-lab`, as shown below. + +![!large error message in small space](../images/globus-uuid.png) + +Similarly, if an invalid bucket name, such as `first_test_bucket`, is entered (due to the use of an underscore, which doesn’t follow LTS bucket naming rules), an error will also be displayed as shown below. To avoid this, please refer to the guidelines for [valid bucket name in LTS](../../storage/lts/lts_faq.md#what-are-valid-bucket-names-in-lts). + +![!large invalid error message in small space](../images/globus_lts_invalid_name_error_001.png) + +```text +Bad Gateway: Endpoint Error, Error (mkdir) +Endpoint: UAB Research Computing LTS (Long Term Storage aka S3) (184408b4-d04b-4513-9912-8feeb6adcab3) +Server: 138.26.220.68:443 +Command: MKD /first_test_bucket/ +Message: Fatal FTP Response --- +Details: 500 +globus_gridftp_server_s3_base: S3 +Error accessing "": ErrorInvalidBucketName: ErrorInvalidBucketName: \r\n +``` + +To cancel or dismiss these errors, click the `refresh list` button on the Globus collection panel, or scroll all the way to the right in the red area of the error message and click the `x` symbol as shown below. + +![!large error message cancellation](../images/globus_lts_cancele_name_error_001.png) + +Uploading a top-level folder that does not follow the bucket naming rules will cause an error similar to the one encountered when creating a bucket with an invalid name. When uploading a folder with a name similar to a bucket in your LTS, Globus will sync and save all sub-folders and files into that bucket. Subfolders also need to follow naming rules, but they inherit the uniqueness from the parent name. This is part of why we recommend random UUIDs. If you attempt to upload a folder with a name that matches an existing bucket in someone else's space, you will encounter a `permission denied error`, as shown below. + +![!large error message in small space](../images/globus_lts_upload_error_001.png) + +Globus can create buckets. By default, buckets are created without a policy, meaning only you can access them until a policy is added. However, Globus cannot be used to modify or add policies. In addition, files transferred to a bucket will become objects with the same name, as long as the name is valid and not duplicated. Globus does not recognize or handle metadata, so you cannot use it to view or modify metadata. For guidance on defining policies for your bucket, please refer to our documentation on [policy structure](../../storage/lts/iam_and_policies.md#policy-structure) and [applying a policy](../../storage/lts/iam_and_policies.md#applying-a-policy). + +## Using Bookmarks in Globus + +To save a bookmark, use the File Manager interface to select a collection and navigate to a path on that collection. Then click the "Create Bookmarks" button as shown below. + +![!Globus File Manager interface with mouse pointer hovering over Bookmark icon.](../images/globus_060_create_bookmark.png) + +To manage bookmarks, click on the Collection search bar, then select the Bookmarks tab. To edit a bookmark, click the 'Pencil' icon. To delete a bookmark, click the 'Trash Bin' icon. + +![!Globus Bookmarks interface showing four bookmarks.](../images/globus_061_manage_bookmarks.png) + + +!!! note + + It is not possible to create bookmarks within High Assurance Endpoints. + diff --git a/docs/data_management/transfer/globus/globus_organization_tutorial.md b/docs/data_management/transfer/globus/globus_organization_tutorial.md new file mode 100644 index 000000000..86cfcc5c2 --- /dev/null +++ b/docs/data_management/transfer/globus/globus_organization_tutorial.md @@ -0,0 +1,265 @@ +# Globus for UAB Research Organizations + +This guide is intended for UAB research managers such as Lab PIs, Core Directors, and their administrative staff. It provides instructions on how to create and search collections and securely share data with collaborators in compliance with security and policy requirements for file sharing. + +If you are new to Globus, we recommend starting with our [Getting Started](./login_to_globus.md) and [Globus for Individual Researchers](../globus/globus_individual_tutorial.md) pages first to familiarize yourself with how Globus is used. When those are complete, we then recommend following the guide on this page in order from start to finish, as each guide assume that the previous ones have been completed. + +Topics covered: + +1. [Why Globus?](#why-globus) +1. [How Does Globus Work?](#how-does-globus-work) +1. [Prerequisites](#prerequisites) +1. [How Do I Get onto the Globus Web App?](#how-do-i-get-onto-the-globus-web-app) +1. [How Do I Find Collections I Created or Own?](#how-do-i-find-collections-i-created-or-own) +1. [How Do I Enable Sharing for My Globus Account?](#how-do-i-enable-collection-sharing-for-my-globus-account) +1. [How Do I Create a Collection?](#how-do-i-create-a-collection) +1. [How Do I Share a Collection with Others?](#how-do-i-share-a-collection-with-others) +1. [How Do I Share Data with a Research Core Customer?](#how-do-i-share-data-with-a-research-core-customer) + +## Why Globus? + +Globus is a data transfer ecosystem that simplifies the process of transferring, sharing, and managing large datasets. It is used by research institutions around the world to move research data between different storage devices, computer systems, and institutions. Globus has many research data oriented features, making it ideal for many research data transfer scenarios. Below is a list of features. + +- Straight-forward, browser-based interface. +- Compatible with [Long-Term Storage](../../storage/lts/index.md). +- Can be used to share data with Research Core customers. +- Can be used to transfer data between lab workstations, servers, and Cheaha. +- Transfers are automatically retried in the event of network or computer system outages. +- Transfers are encrypted end-to-end. Globus never sees your data. +- Suitable for transferring PHI and HIPAA data. Note: a UAB Enterprise IT risk assessment is required. + +## How Does Globus Work? + +Globus is an ecosystem of software intended to make research data transfer simpler. The Globus Web Application (Web App) at allows you to initiate transfers between any two Collections you have authorization to access. The Globus Connect Personal (GCP) and Globus Connect Server (GCS) software let you turn any computer into a Globus Collection. At no point do Globus servers touch your research data. Instead, when you initiate a transfer between two Collections, the Globus application tells the two Collections that they need to talk to each other and data is sent directly between them. The Collections update the application with information you may need to know, such as how much data has transferred so far, how fast the transfer is proceeding, and any errors that occur. If the connection between Collections is interrupted for any reason, the Globus application will attempt to restart the transfer from where it left off. + +## Prerequisites + +For these guides, you will need your BlazerID or [XIAS ID](../../../account_management/xias/index.md) and password to authenticate using UAB Single Sign-On (SSO). + +## How Do I Get Onto the Globus Web App? + +To learn how to get onto the Globus Web App [Getting Started](./login_to_globus.md#how-do-i-get-onto-the-globus-web-app). Please visit that link and then return here when you have finished. + +## How Do I Find Collections I Created or Own? + +To find a Collection you own, use the following steps. + +1. Navigate to the [Globus Web App](#how-do-i-get-onto-the-globus-web-app) using your browser. You should be at the File Manager page. + + ![File Manager page of the Globus Web App](../images/common/file-manager.png) + +1. Click either of the Collection Search bar at the top of the File Manager page. This will take you to the Collection Search page. + + ![Collection Search page of the Globus Web App](../images/common/collection-search-page.png) + +1. Click the Your Collections tab to display a list of Collections you have created or own. + + ![Your Collections tab showing a GCP Collection.](../images/go-my-collections/003-your-collections.png) + +1. From here there are two options: + + 1. Click the name of the Collection to select it for a file transfer and be taken back to the File Manager page. + + ![File Manager page showing selected Collection on left-hand panel.](../images/go-my-collections/004-file-manager-selection-made.png) + + 1. Click the three dots icon at the right side of the entry to be taken to the Collection details page. + + ![Collectionm details page with overview tab selected.](../images/go-my-collections/005-collection-details-page.png) + +## How Do I Enable Collection Sharing for My Globus Account? + +Before you can share Collections from your Globus Connect Personal (GCP) Collection with others, you must do a one-time setup for your account. You will need to join the "University of Alabama at Birmingham (HA)" (UAB HA) subscription group. Sharing any Collection requires a paid subscription with Globus. UAB Research Computing has a subscription, but Globus does not know your BlazerID is part of our subscription until you join the subscription group. So, Globus also does not know the GCP Collection you created is part of our subscription. By joining our UAB HA group, you and your GCP Collection are confirmed to be part of our subscription, and you can share Collections from the GCP Collection. + +To join the UAB HA group, we need to receive both a UAB support request, and a request to join the UAB HA group within the Globus Web App. We need both because we sometimes get spam applications through Globus. Having a Support Request helps us filter the spam. Please use the following steps to join. + +1. Submit a [Support Request](../../../help/support.md#how-do-i-create-a-support-ticket). In the request please include the following. + + - Your BlazerID. + - The text "Please add me to the Globus UAB HA subscription group." + - The reason you need to be able to share a Collection in Globus. For Research Cores, this would be to share data with your customers. + +1. In your browser [get onto the Globus Web App](#how-do-i-get-onto-the-globus-web-app). +1. In the left hand navigation panel click "Groups" to be taken to the Groups page. + + ![Globus groups page with no groups.](../images/go-uab-ha-group-access/001-groups-page.png) + +1. Ensure the "MY GROUPS" box is unchecked. In the "Filter groups" search bar enter "University of Alabama at Birmingham" to locate the "University of Alabama at Birmingham (HA)" (UAB HA) group. + + ![Globus group page showing search results with University of Alabama at Birmingham (HA) listed](../images/go-uab-ha-group-access/002-groups-search.png) + +1. Click on the UAB HA group name to be taken to the group details page. The Overview tab should be selected. + + ![UAB HA subscription group details page with overview tab selected.](../images/go-uab-ha-group-access/003-uab-ha-overview.png) + +1. Click on the "Join this Subscription" button to be taken to the form to submit a request to join. + + ![UAB HA subscription group join request form.](../images/go-uab-ha-group-access/004-join-form.png) + +1. Fill in the form fields and click the "Submit Application" button when completed. This will send you to a page notifying you that your membership is pending. A request has been sent to Research Computing, so please wait until you see a reply in the support request. + + ![Notice of pending membership.](../images/go-uab-ha-group-access/005-pending.png) + +1. When your membership has been accepted, you can verify by [returning to the Globus Web App](#how-do-i-get-onto-the-globus-web-app) and navigating to the Groups page. Check the "MY GROUPS". You should see "University of Alabama at Birmingham (HA)" listed with a green, circled checkmark and the word "ACTIVE". + + ![Groups page with my groups checked showing UAB HA subscription group](../images/go-uab-ha-group-access/006-my-groups.png.png) + +When you have completed the steps in this section, proceed to [creating a Collection](#how-do-i-create-a-collection). + +## How Do I Create a Collection? + +There are three ways to create a Collection. + +- Start from [an existing Collection you created or own](#how-do-i-find-collections-i-created-or-own). +- [Install Globus Connect Personal](../globus/gcp_install.md#how-to-install-globus-connect-personal-gcp) and [share a folder](../globus/gcp_setup.md#how-do-i-choose-specific-folders-to-share-using-globus-connect-personal) to create a Collection. +- Install and configure one or more Collections with Globus Connect Server. To do this, please [Contact Support](../../../help/support.md#how-do-i-create-a-support-ticket) to start a discussion. + +The instructions below assume you are starting from an existing Collection. The instructions will work to create a subset of your Globus Connect Personal Collection. + +1. [Get onto the Globus Web App](#how-do-i-get-onto-the-globus-web-app). +1. [Find the existing Collection](#how-do-i-find-collections-i-created-or-own) and access its details page. +1. Click the "Collection" tab. + + ![Overview tab of details page with Collection tab to the right.](../images/go-create-collection/001-collection-tab.png) + +1. Click the "+ Add Guest Collection" button to be taken to a form to create the new Collection. + + ![Collection tab of details page with Add Guest Collection button.](../images/go-create-collection/002-add-guest-collection.png) + +1. Fill out the form. + + - **(1) You Are Sharing**: Shows which Collection you will be sharing all or part of. Helpful to verify you are sharing the right Collection + - **(2) Path**: Which path you wish to share. By default, the "/" path is the root of the original Collection. This may not be appropriate for your use case. Generally, when creating one Collection from another, you will want to pick a subfolder of the original Collection. Do this with the "Browse" button. + - **(3) Display Name**: The name people will see when accessing this Collection, and what they will search for when looking for the Collection. + - **(4) Description**: An optional description for the Collection. + - **(5) Keywords**: Optional keywords to help locating the Collection. May be useful for broadly or publicly shared data. + + ![Collection creation form.](../images/go-create-collection/003-form.png) + +1. Click the "Create Guest Collection" button to create the Collection. You will be taken to the details page of the new Collection, on the Permissions tab. + + ![Permissions tab of new Collection details page.](../images/go-create-collection/004-completion.png) + +When you have created a Collection, you are ready to [share the Collection with others](#how-do-i-share-a-collection-with-others). + +## How Do I Share a Collection With Others? + +Before sharing a Collection with others, you will need to first [create a Collection](#how-do-i-create-a-collection) you administer. If the Collection is a [Globus Connect Personal Collection](../globus/gcp_install.md#how-to-install-globus-connect-personal-gcp), then you will also need to [enable Collection sharing for your Globus account](#how-do-i-enable-collection-sharing-for-my-globus-account). If these prerequisites have been met, then you are ready to setup a Collection to be shared with others. Please follow the instructions below. + +1. [Get onto the Globus Web App](#how-do-i-get-onto-the-globus-web-app). +1. [Find the Collection](#how-do-i-find-collections-i-created-or-own) you wish to share. In this case we are looking for the "customer-share" Collection. +1. On the "customer-share" Collection details page, click the Permissions tab. + + ![Collection details page overview tab.](../images/go-share-collection/001-collection-details-overview.png) + +1. Click the "Add Permissions – Share With" button. + + ![Collection details page permissions tab.](../images/go-share-collection/002-collection-details-permissions-before.png) + +1. Fill in the form. + + - **Path** defaults to the root of the Collection. This may or may not be acceptable for your use case. Be sure to limit access to only the data that needs to be shared. Use the file browser available in the form to find the correct level of access. For now grant permission to the default. + - **Share With**: radio buttons to provide control over who to share with. In almost all cases, you will want to share with a single user, which is what we will show here. + - **User**: Use the "User" search bar to find the specific user you wish to share with. If they have never accessed Globus before, they will not appear in the search results. You can safely enter their email address to add them anyway. For now grant permission to yourself for the purposes of experimenting. + - **Email Notification** checkbox. We recommend sending an email notification as a convenience. + - **To** entry field: who to send the email to. We recommend the same person as the "User" selected earlier. There may be future cases where you want to notify others, such as a supervisor, as well. + - **Message** text entry field: the optional content to send in the email message. + - **Permissions** The "read" permission must be granted, as that is the point of sharing the Collection. You may additionally give "write" permission to create a two-way collaboration. We recommend Research Cores not grant "write" permission. If you are using a Globus Connect Personal Collection, then "write" permission requires you to correctly [configure your Collection](../globus/gcp_setup.md#how-do-i-choose-specific-folders-to-share-using-globus-connect-personal) to make your shared folder writable. + + ![Add permissions form.](../images/go-share-collection/003-add-permissions-form.png) + +1. Click the "Add Permission" button to grant permission. You should see a notification confirming the permissions granted. At this point permissions have been granted and the Collection is shared with another person. If you need to add more people, click the "Add another Permision" button and repeat the process. Otherwise click "Done". For now click "Done". + + ![Permission confirmation notification.](../images/go-share-collection/004-permissions-confirmation.png) + +1. When you click "Done" you should be taken back to the Permissions tab of the Customer Share page. You should see a new entry with "Path: /". If you click the drop-down arrow you will see yourself listed with "Read" permission. If you need to revoke permissions, return to this page and click the icon that looks like a trash can. + + ![Collection details page permissions tab showing new entry.](../images/go-share-collection/005-colection-details-permissions-after.png) + +## How Do I Share Data With a Research Core Customer? + +If you manage data for a Research Core, please [Contact Support](../../../help/support.md#how-to-request-support) to start a conversation. The answer to this question is currently being developed, so we will need to work together to find the ideal solution. + +## Managing Guest Collections From a Globus Endpoint + +It is NOT RECOMMENDED to make Globus Connect Personal Guest Collections public as this is insecure. It is more difficult to manage access controls for the entire Globus Connect Personal Collection than for a Guest Collection. Guest collections make it simpler to share different data with distinct collaborators, and to manage who has access to what data. Be secure, use Guest Collections! + +### Creating a Guest Collection + +1. Click "Collections" in the left-hand navigation pane. + +1. Click the "Administered By You" tab (or) you can search for the collections in search. + + ![!Globus Endpoints page with Administered by You selected, showing two endpoints. One of the endpoints is a shared endpoint.](../images/globus_100_shared_search_collections.png) + +1. In the table, find the collections you wish to share data from and click its name. You will be taken to the page for that collection. + +1. Click the "Collections" tab. + + ![!Globus UAB RC Work Laptop page with Guest Collections tab selected showing one collection.](../images/globus_101_shared_collections.png) + +1. Click the "Add a Guest Collection" button. + +1. Fill out the form. + + ![!Create New Guest Collection form.](../images/globus_102_shared_collection_form.png) + + 1. Manually enter a path or click the Browse button to select a folder. + 1. Give a short but memorable name for your Guest Collection. This information will be useful for your collaborators. + 1. Optionally fill in a more detailed description of the Guest Collection for your records. + 1. Optionally fill in searchable keywords. + 1. Other additional options include, information link, contact email, organization/department. + 1. Default directory, if left empty, is equivalent to the first field "Directory". + +1. Click "Create Collection" to move to the next step. You will be taken to the page for the newly created collection, which is now a full-fledged Guest Collection. Any further references to "an endpoint" will be about the newly created, Guest Collection. + +1. Make sure you are on the "Permissions" tab. You should see a permissions table with your name in it. + + ![!Newly created test endpoint page with Permissions tab selected.](../images/globus_103_shared_permissions.png) + +1. Click "Add Permissions -- Share With" to share your Guest Collection with other users. + +1. Fill out the form. + + ![!Test endpoint Add Permissions Share With form.](../images/globus_104_shared_add_permissions.png) + + 1. Optionally enter a path within the shared endpoint or use the Browse button. If you leave the path as just a slash, the entire shared endpoint will be shared with the permitted users. + 1. Select who to share with. + 1. User - One or more users. + 1. Group - All members of a group. + 1. All Users - All globus users. + 1. Public - Makes data accessible to everyone. + + + !!! danger + + It is important to note that options (iii) and (iv) poses a high risk in terms of security. Therefore, we strongly advise against this practice.This will expose information to everyone on Globus! + + + 1. Search for users to add, or a group, depending on your choice above. You should be able to find any globus user using the search box. + + + !!! warning + + Be certain of which user you are selecting! Check the email address domain. + + + 1. If adding users, optionally enter a message so they know why they are being added. + 1. Select permissions. Read is automatically selected and cannot be changed. Write permissions are optional. + +1. Click "Add Permission" to add permissions for these users or groups. You will be returned to the page for the Guest Collection and should be on the "Permissions" tab and should see the user or group in the table. + +### Deleting a Guest Collection + +1. Click "Collections" in the left-hand navigation pane, then + +1. Click the "Administered By You" tab. + + ![!Globus Endpoints page with Administered by You tab selected, showing two endpoints.](../images/globus_105_shared_collections.png) + +1. Click the right caret ">" icon at the right side of the row with the collection you wish to delete. You will be taken to the information page for that collection. + + ![!Globus Endpoints page with Shared Collection Overview.](../images/globus_106_shared_collection_overview.png) + +1. Click "X Delete Collection" and a confirmation dialog will open at the top of the page. Respond to the dialog to delete the Guest Collection, or to cancel. + + ![!Delete Collection confirmation dialog banner.](../images/globus_107_shared_delete.png) diff --git a/docs/data_management/transfer/globus/index.md b/docs/data_management/transfer/globus/index.md new file mode 100644 index 000000000..f63775a35 --- /dev/null +++ b/docs/data_management/transfer/globus/index.md @@ -0,0 +1,51 @@ +# Globus + + +!!! announcement + + We are aware of reduced transfer speed into and out of [Long-Term Storage (LTS)](../../storage/lts/index.md) when using Globus and are working on a solution. For some use cases, [`s5cmd`]( ../../storage/lts/interfaces.md#s5cmd) may be a faster alternative. Please see [Our News Section](../../../news/posts/2025-04-07-reduced-lts-transfer-speeds-on-globus.md) for more information. + + Access to Cheaha storage (GPFS) is now available through the new Globus mapped collection "**UAB RCS Cheaha HPC**" for both on-campus and off-campus connections. The new mapped collection replaces the previous "Cheaha cluster on-campus (UAB Science DMZ)" and "Cheaha cluster off-campus (UAB Science DMZ)" mapped collections. This change comes from our recently completed GPFS5 migration and unifies access to Cheaha storage via Globus. + + +Globus is a powerful tool for robustly and securely managing data transfers to and from collaborators and within UAB Research Computing. Globus is a GUI-based application and recommended for most single-use and day-to-day data transfer needs. + + +!!! note + + UAB Research Computing uses High Assurance Endpoints, meaning there are additional security measures in place to reduce risk and move toward HIPAA compliance. Generally speaking, if you have used Globus in the past, the data transfer interface has not changed, but there are a few new restrictions/changes. + + 1. You will be prompted to prove authorization each time you access a UAB Research Computing endpoint or attempt to download files to your local machine from such an endpoint. If you are already logged in with Single Sign-On (SSO) the process is simple. If not, you will need to authenticate with SSO. + 1. Bookmarks are not allowed in High Assurance endpoints. + + For more detailed information on High Assurance please see the Globus official pages below: + + - [High Assurance Security Overview](https://docs.globus.org/guides/overviews/security/high-assurance-overview/) + - [High Assurance Collections](https://docs.globus.org/guides/overviews/high-assurance/) + + +## Globus Endpoints + +The following Globus collection (endpoints) are currently available and supported by UAB Research Computing for transferring data to and from collections within the UAB Research Computing storage systems: + +- Remote Globus Collection (Endpoints): UAB Research Computing currently supports the following three remote Globus endpoints. + - UAB Research Computing LTS (Long Term Storage aka S3): Access data on [Long Term Storage (LTS)](../../storage/lts/index.md). + - UAB RCS Cheaha HPC: Access data on GPFS, including [User Data and Home directories](../../storage/cheaha_storage_gpfs/individual_directories.md), [Scratch](../../storage/cheaha_storage_gpfs/network_scratch.md), and [Project Directories](../../storage/cheaha_storage_gpfs/project_directories.md). + - UAB Box: Access data on UAB Box. + +- Local (Personal) Endpoints: These are Globus collections that you can create on your own computer (laptops, desktops, workstations, and other self-managed systems) with [Globus Connect Personal (GCP)](./gcp_setup.md). + + +!!! important + + To use UAB Research Computing endpoints, you will need to ensure you are using your UAB identity. + + +Refer to the topics below to learn how to get started with Globus and use its key features. + +- [Getting Started](./login_to_globus.md): Access the Globus Web App and login with UAB credentials. +- [Install Globus Connect Personal (GCP)](./gcp_install.md): Turn your personal computer into a Globus endpoint. Only specific folders are accessible by default. +- [Set up and use Globus Connect Personal (GCP)](./gcp_setup.md): Log in, configure your endpoint, select folders, and start transferring files. +- [Globus for Individuals](./globus_individual_tutorial.md): Using Globus as a researcher for personal data transfers. +- [Globus for Organizations (Labs and Cores)](./globus_organization_tutorial.md): Shared endpoints for labs and research cores, with centralized management +- [Globus for Group Management](./globus_group_management.md): Manage access to data via teams and project groups. diff --git a/docs/data_management/transfer/globus/login_to_globus.md b/docs/data_management/transfer/globus/login_to_globus.md new file mode 100644 index 000000000..6dae757db --- /dev/null +++ b/docs/data_management/transfer/globus/login_to_globus.md @@ -0,0 +1,85 @@ + +# Logging in to Globus + +UAB has a subscription to Globus,allowing you to set up a Globus account using your UAB credentials. Follow the steps below to log in and complete your Globus account setup. + +## How Do I Get Onto the Globus Web App? + +1. Use your browser to navigate to . You should see a login page similar to the one shown below. Click "LOG IN". + + ![Globus Web App login page.](../images/gi-web-app/001-login.png) + +1. After clicking "LOG IN", the organization search page will appear, as shown below. + + ![Globus organization search page.](../images/gi-web-app/002-login.png) + +1. Find and select our institution. Type "UAB" or "University of Alabama at Birmingham" into the search bar to locate UAB in the list. The image below shows the correct choice in a red box. + + + !!! note + + If you are an external collaborator using an [XIAS account](../../../account_management/xias/index.md) (this is uncommon) to access UAB-owned storage, do not use your home institution login. You will only be able to access UAB storage systems using your XIAS credentials. + + XIAS accounts are not currently able to access XD storage via Globus. Please use [Other Data Transfer Options](../index.md#data-management-and-transfer-tools) for supported alternatives. + + + ![Globus Web App login search bar with UAB entered and University of Alabama at Birmingham in a red box.](../images/gi-web-app/002-search.png) + +1. Select "University of Alabama at Birmingham" from the drop-down menu. + + ![Globus Web App with University of Alabama at Birmingham selected showing activated Continue button.](../images/gi-web-app/003-select.png) + +1. Click "Continue" to be taken to the UAB Single Sign-On (SSO) form. Enter your BlazerID and password in the SSO form, then click "Log In". Complete the login process as usual. + + ![UAB Single Sign-On form.](../images/gi-web-app/004-sso.png) + +1. You should be taken to the File Manager page of the Globus Web App. We will be revisiting this page frequently throughout the tutorials. We highly recommend taking some time to familiarize yourself with the page as you proceed. The next few steps outline the important features of the File Manager page and its purpose. + + ![Globus Web App file manager single-panel view.](../images/gi-web-app/005-file-manager-single-panel.png) + +1. Recall that Globus is a data transfer application. A data transfer means moving data between two computers: source and destination. As we learn more about Globus, data stored on the source and destination will be visible on the File Manager page in the panels in the screenshot above. + + By default, only one panel is visible. We recommend selecting two-panel view mode for improved ease of use. Select two-panel view mode by clicking the button located near the top right corner, as shown in the red box below. Of the three available view mode buttons, the two-panel view mode button is in the center. + + From here on, tutorials will assume you are using two-panel view mode when we refer to the File Manager page. + + ![Globus Web App file manager two-panel view.](../images/gi-web-app/006-view-buttons.png) + +1. Now that two-panel view mode has been selected, you will see two sets of features and panels side-by-side. Most of these features will be used extensively when using Globus. + + - **(1) Collection Search bar**: Clicking here will open the Collection Search page, allowing you to find Collections of data shared by others, or find your own Collections. The next step will demonstrate features of the Collection Search page. + - **(2) Path text field**: After a Collection has been selected using (1), this text field allows you to enter the path to a specific folder, updating the files and folders shown in (5). + - **(3) Start Transfer button**: When you have selected files and folders in (5), use this button to start transferring. + - **(4) Transfer Options drop down menu**: Various options for the transfer are available in this menu. These options are not commonly needed and are not used in the tutorials. + - **(5) File Browser panel**: After a Collection has been selected in (1), a list of files and folders will appear here for the path shown in (2). You may navigate the Collection here like you would on your operating system. + - **(6) Navigation panel**: The blue bar at the left hand side of the Globus Web App is the navigation panel. It will be referred to and used many times throughout these tutorials. + + ![Globus Web App file manager two-panel view.](../images/gi-web-app/007-file-manager-two-panel.png) + +1. To better prepare you for what to expect, the screenshot below shows the File Manager page with an example Collection selected, called "my-gcp-collection". This example shows a path in the Path text field and contents in the File Browser panel. + + - **(1) Select All checkbox**: Select or unselect all files in the currently displayed folder. + - **(2) Go Up One Level button**: Go to the folder containing the currently displayed folder. + - **(3) Refresh button**: Refreshes the view of the currently displayed folder. + - **(4) File Selection checkbox**: If checked, then the file is selected for transfer. This file is selected. + - **(5) Folder Selection checkbox**: If checked, then the folder is currently selected for transfer. This folder is not selected. + + ![Globus Web App file manager two-panel view with Collection selected showing a file and folder.](../images/gi-web-app/008-file-manager-two-panel-example.png) + +The File Manager page will be your most frequently-visited page when using Globus for data transfers. It is central to usage of the Globus Web Application. Please take some time to familiarize yourself with its look and feel. As you progress in the tutorials, please take time to experiment with transferring data to better understand how the interface works. Feel free to return here for guidance. + +From here you can proceed to [How Do I Search for Collections by Name?](./globus_individual_tutorial.md#how-do-i-search-for-collections-by-name) and [How Do I Transfer Data Between Collections?](./globus_individual_tutorial.md#how-do-i-transfer-data-between-collections) + +## Managing Identity + +Globus Identity is a concept helping to map Globus Accounts (one per person) to institutions (one or more per person). Most UAB researchers will have a single identity, their UAB identity, tied to their BlazerID. Some researchers may have external collaborations or appointments that provide additional entities which need access to other endpoints on Globus. + +To manage your identities, navigate to and sign in. + +## Where Can I Learn More About Globus? + +- Visit the [Globus Events Calendar](https://www.globus.org/events) for official learning opportunities. + +At this point, you can already start using Globus through your browser to move data between existing endpoints. + +However, if you want to move data to or from your own computer, you will need to install Globus Connect Personal. The [Install Globus Connect Personal](./gcp_install.md) and [Set Up Globus Connect Personal](./gcp_setup.md) walk you through that process. diff --git a/docs/data_management/transfer/tutorial/res/globus-group-roles.csv b/docs/data_management/transfer/globus/res/globus-group-roles.csv similarity index 100% rename from docs/data_management/transfer/tutorial/res/globus-group-roles.csv rename to docs/data_management/transfer/globus/res/globus-group-roles.csv diff --git a/docs/data_management/transfer/tutorial/images/common/collection-search-page.png b/docs/data_management/transfer/images/common/collection-search-page.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/common/collection-search-page.png rename to docs/data_management/transfer/images/common/collection-search-page.png diff --git a/docs/data_management/transfer/tutorial/images/common/file-manager.png b/docs/data_management/transfer/images/common/file-manager.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/common/file-manager.png rename to docs/data_management/transfer/images/common/file-manager.png diff --git a/docs/data_management/transfer/tutorial/images/common/groups-checked.png b/docs/data_management/transfer/images/common/groups-checked.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/common/groups-checked.png rename to docs/data_management/transfer/images/common/groups-checked.png diff --git a/docs/data_management/transfer/tutorial/images/common/groups-unchecked.png b/docs/data_management/transfer/images/common/groups-unchecked.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/common/groups-unchecked.png rename to docs/data_management/transfer/images/common/groups-unchecked.png diff --git a/docs/data_management/transfer/tutorial/images/common/groups.png b/docs/data_management/transfer/images/common/groups.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/common/groups.png rename to docs/data_management/transfer/images/common/groups.png diff --git a/docs/data_management/transfer/tutorial/images/common/ha-authenticate.png b/docs/data_management/transfer/images/common/ha-authenticate.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/common/ha-authenticate.png rename to docs/data_management/transfer/images/common/ha-authenticate.png diff --git a/docs/data_management/transfer/tutorial/images/common/select-identity.png b/docs/data_management/transfer/images/common/select-identity.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/common/select-identity.png rename to docs/data_management/transfer/images/common/select-identity.png diff --git a/docs/data_management/transfer/tutorial/images/gg-accept-group/001-pending.png b/docs/data_management/transfer/images/gg-accept-group/001-pending.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-accept-group/001-pending.png rename to docs/data_management/transfer/images/gg-accept-group/001-pending.png diff --git a/docs/data_management/transfer/tutorial/images/gg-accept-group/002-buttons.png b/docs/data_management/transfer/images/gg-accept-group/002-buttons.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-accept-group/002-buttons.png rename to docs/data_management/transfer/images/gg-accept-group/002-buttons.png diff --git a/docs/data_management/transfer/tutorial/images/gg-accept-group/003-form.png b/docs/data_management/transfer/images/gg-accept-group/003-form.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-accept-group/003-form.png rename to docs/data_management/transfer/images/gg-accept-group/003-form.png diff --git a/docs/data_management/transfer/tutorial/images/gg-create-group/001-create-group-button.png b/docs/data_management/transfer/images/gg-create-group/001-create-group-button.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-create-group/001-create-group-button.png rename to docs/data_management/transfer/images/gg-create-group/001-create-group-button.png diff --git a/docs/data_management/transfer/tutorial/images/gg-create-group/002-create-group-form.png b/docs/data_management/transfer/images/gg-create-group/002-create-group-form.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-create-group/002-create-group-form.png rename to docs/data_management/transfer/images/gg-create-group/002-create-group-form.png diff --git a/docs/data_management/transfer/tutorial/images/gg-create-group/003-group-details-page.png b/docs/data_management/transfer/images/gg-create-group/003-group-details-page.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-create-group/003-group-details-page.png rename to docs/data_management/transfer/images/gg-create-group/003-group-details-page.png diff --git a/docs/data_management/transfer/tutorial/images/gg-create-subgroup/001-subgroup-tab.png b/docs/data_management/transfer/images/gg-create-subgroup/001-subgroup-tab.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-create-subgroup/001-subgroup-tab.png rename to docs/data_management/transfer/images/gg-create-subgroup/001-subgroup-tab.png diff --git a/docs/data_management/transfer/tutorial/images/gg-create-subgroup/002-create-subgroup.png b/docs/data_management/transfer/images/gg-create-subgroup/002-create-subgroup.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-create-subgroup/002-create-subgroup.png rename to docs/data_management/transfer/images/gg-create-subgroup/002-create-subgroup.png diff --git a/docs/data_management/transfer/tutorial/images/gg-delete-group/001-delete-button.png b/docs/data_management/transfer/images/gg-delete-group/001-delete-button.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-delete-group/001-delete-button.png rename to docs/data_management/transfer/images/gg-delete-group/001-delete-button.png diff --git a/docs/data_management/transfer/tutorial/images/gg-find-group/003-search-bar.png b/docs/data_management/transfer/images/gg-find-group/003-search-bar.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-find-group/003-search-bar.png rename to docs/data_management/transfer/images/gg-find-group/003-search-bar.png diff --git a/docs/data_management/transfer/tutorial/images/gg-find-group/004-results.png b/docs/data_management/transfer/images/gg-find-group/004-results.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-find-group/004-results.png rename to docs/data_management/transfer/images/gg-find-group/004-results.png diff --git a/docs/data_management/transfer/tutorial/images/gg-find-group/005-group-details-page.png b/docs/data_management/transfer/images/gg-find-group/005-group-details-page.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-find-group/005-group-details-page.png rename to docs/data_management/transfer/images/gg-find-group/005-group-details-page.png diff --git a/docs/data_management/transfer/tutorial/images/gg-invite-to-group/001-invite-button.png b/docs/data_management/transfer/images/gg-invite-to-group/001-invite-button.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-invite-to-group/001-invite-button.png rename to docs/data_management/transfer/images/gg-invite-to-group/001-invite-button.png diff --git a/docs/data_management/transfer/tutorial/images/gg-invite-to-group/002-search-and-add.png b/docs/data_management/transfer/images/gg-invite-to-group/002-search-and-add.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-invite-to-group/002-search-and-add.png rename to docs/data_management/transfer/images/gg-invite-to-group/002-search-and-add.png diff --git a/docs/data_management/transfer/tutorial/images/gg-invite-to-group/003-select-role.png b/docs/data_management/transfer/images/gg-invite-to-group/003-select-role.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-invite-to-group/003-select-role.png rename to docs/data_management/transfer/images/gg-invite-to-group/003-select-role.png diff --git a/docs/data_management/transfer/tutorial/images/gg-invite-to-group/004-confirmation.png b/docs/data_management/transfer/images/gg-invite-to-group/004-confirmation.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-invite-to-group/004-confirmation.png rename to docs/data_management/transfer/images/gg-invite-to-group/004-confirmation.png diff --git a/docs/data_management/transfer/tutorial/images/gg-join-group/001-general-page-button.png b/docs/data_management/transfer/images/gg-join-group/001-general-page-button.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-join-group/001-general-page-button.png rename to docs/data_management/transfer/images/gg-join-group/001-general-page-button.png diff --git a/docs/data_management/transfer/tutorial/images/gg-join-group/001-specific-page-button.png b/docs/data_management/transfer/images/gg-join-group/001-specific-page-button.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-join-group/001-specific-page-button.png rename to docs/data_management/transfer/images/gg-join-group/001-specific-page-button.png diff --git a/docs/data_management/transfer/tutorial/images/gg-join-group/002-form.png b/docs/data_management/transfer/images/gg-join-group/002-form.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-join-group/002-form.png rename to docs/data_management/transfer/images/gg-join-group/002-form.png diff --git a/docs/data_management/transfer/tutorial/images/gg-join-group/003-pending.png b/docs/data_management/transfer/images/gg-join-group/003-pending.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-join-group/003-pending.png rename to docs/data_management/transfer/images/gg-join-group/003-pending.png diff --git a/docs/data_management/transfer/tutorial/images/gg-leave-group/001-leave-button.png b/docs/data_management/transfer/images/gg-leave-group/001-leave-button.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-leave-group/001-leave-button.png rename to docs/data_management/transfer/images/gg-leave-group/001-leave-button.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-group-hide-group/001-hide-group-radio-button.png b/docs/data_management/transfer/images/gg-modify-group-hide-group/001-hide-group-radio-button.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-group-hide-group/001-hide-group-radio-button.png rename to docs/data_management/transfer/images/gg-modify-group-hide-group/001-hide-group-radio-button.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-group-hide-members/001-hide-members-radio-button.png b/docs/data_management/transfer/images/gg-modify-group-hide-members/001-hide-members-radio-button.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-group-hide-members/001-hide-members-radio-button.png rename to docs/data_management/transfer/images/gg-modify-group-hide-members/001-hide-members-radio-button.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-group-invite-only/001-request-to-join-checkbox.png b/docs/data_management/transfer/images/gg-modify-group-invite-only/001-request-to-join-checkbox.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-group-invite-only/001-request-to-join-checkbox.png rename to docs/data_management/transfer/images/gg-modify-group-invite-only/001-request-to-join-checkbox.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-group/001-settings-tab.png b/docs/data_management/transfer/images/gg-modify-group/001-settings-tab.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-group/001-settings-tab.png rename to docs/data_management/transfer/images/gg-modify-group/001-settings-tab.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-members-remove/002-membership-page.png b/docs/data_management/transfer/images/gg-modify-members-remove/002-membership-page.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-members-remove/002-membership-page.png rename to docs/data_management/transfer/images/gg-modify-members-remove/002-membership-page.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-members-remove/003-membership-page-after.png b/docs/data_management/transfer/images/gg-modify-members-remove/003-membership-page-after.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-members-remove/003-membership-page-after.png rename to docs/data_management/transfer/images/gg-modify-members-remove/003-membership-page-after.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-members-role/003-pencil-edit.png b/docs/data_management/transfer/images/gg-modify-members-role/003-pencil-edit.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-members-role/003-pencil-edit.png rename to docs/data_management/transfer/images/gg-modify-members-role/003-pencil-edit.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-members-role/004-role-controls.png b/docs/data_management/transfer/images/gg-modify-members-role/004-role-controls.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-members-role/004-role-controls.png rename to docs/data_management/transfer/images/gg-modify-members-role/004-role-controls.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-members/001-members-tab.png b/docs/data_management/transfer/images/gg-modify-members/001-members-tab.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-members/001-members-tab.png rename to docs/data_management/transfer/images/gg-modify-members/001-members-tab.png diff --git a/docs/data_management/transfer/tutorial/images/gg-modify-members/002-membership-page.png b/docs/data_management/transfer/images/gg-modify-members/002-membership-page.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-modify-members/002-membership-page.png rename to docs/data_management/transfer/images/gg-modify-members/002-membership-page.png diff --git a/docs/data_management/transfer/tutorial/images/gg-view-members/001-members-tab-page.png b/docs/data_management/transfer/images/gg-view-members/001-members-tab-page.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-view-members/001-members-tab-page.png rename to docs/data_management/transfer/images/gg-view-members/001-members-tab-page.png diff --git a/docs/data_management/transfer/tutorial/images/gg-view-subgroups/001-subgroup-tab-page.png b/docs/data_management/transfer/images/gg-view-subgroups/001-subgroup-tab-page.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gg-view-subgroups/001-subgroup-tab-page.png rename to docs/data_management/transfer/images/gg-view-subgroups/001-subgroup-tab-page.png diff --git a/docs/data_management/transfer/tutorial/images/gi-check-transfer-status/001-activity.png b/docs/data_management/transfer/images/gi-check-transfer-status/001-activity.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-check-transfer-status/001-activity.png rename to docs/data_management/transfer/images/gi-check-transfer-status/001-activity.png diff --git a/docs/data_management/transfer/tutorial/images/gi-check-transfer-status/002-status.png b/docs/data_management/transfer/images/gi-check-transfer-status/002-status.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-check-transfer-status/002-status.png rename to docs/data_management/transfer/images/gi-check-transfer-status/002-status.png diff --git a/docs/data_management/transfer/tutorial/images/gi-find-shared-collections/003-list.png b/docs/data_management/transfer/images/gi-find-shared-collections/003-list.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-find-shared-collections/003-list.png rename to docs/data_management/transfer/images/gi-find-shared-collections/003-list.png diff --git a/docs/data_management/transfer/images/gi-find-shared-collections/004-selected.png b/docs/data_management/transfer/images/gi-find-shared-collections/004-selected.png new file mode 100644 index 000000000..05e2a2b7a Binary files /dev/null and b/docs/data_management/transfer/images/gi-find-shared-collections/004-selected.png differ diff --git a/docs/data_management/transfer/tutorial/images/gi-search-collections/001-search-bar.png b/docs/data_management/transfer/images/gi-search-collections/001-search-bar.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-search-collections/001-search-bar.png rename to docs/data_management/transfer/images/gi-search-collections/001-search-bar.png diff --git a/docs/data_management/transfer/tutorial/images/gi-search-collections/002-collection-search.png b/docs/data_management/transfer/images/gi-search-collections/002-collection-search.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-search-collections/002-collection-search.png rename to docs/data_management/transfer/images/gi-search-collections/002-collection-search.png diff --git a/docs/data_management/transfer/tutorial/images/gi-search-collections/003-results.png b/docs/data_management/transfer/images/gi-search-collections/003-results.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-search-collections/003-results.png rename to docs/data_management/transfer/images/gi-search-collections/003-results.png diff --git a/docs/data_management/transfer/images/gi-transfer-collection-to-cheaha/005-ready-to-select.png b/docs/data_management/transfer/images/gi-transfer-collection-to-cheaha/005-ready-to-select.png new file mode 100644 index 000000000..bb118e378 Binary files /dev/null and b/docs/data_management/transfer/images/gi-transfer-collection-to-cheaha/005-ready-to-select.png differ diff --git a/docs/data_management/transfer/images/gi-transfer-collection-to-cheaha/008-selection.png b/docs/data_management/transfer/images/gi-transfer-collection-to-cheaha/008-selection.png new file mode 100644 index 000000000..130618b07 Binary files /dev/null and b/docs/data_management/transfer/images/gi-transfer-collection-to-cheaha/008-selection.png differ diff --git a/docs/data_management/transfer/images/gi-transfer-collection-to-cheaha/009-transfer-started.png b/docs/data_management/transfer/images/gi-transfer-collection-to-cheaha/009-transfer-started.png new file mode 100644 index 000000000..3fb80e0c2 Binary files /dev/null and b/docs/data_management/transfer/images/gi-transfer-collection-to-cheaha/009-transfer-started.png differ diff --git a/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-lts/005-ready-to-select.png b/docs/data_management/transfer/images/gi-transfer-collection-to-lts/005-ready-to-select.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-lts/005-ready-to-select.png rename to docs/data_management/transfer/images/gi-transfer-collection-to-lts/005-ready-to-select.png diff --git a/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-lts/008-selection.png b/docs/data_management/transfer/images/gi-transfer-collection-to-lts/008-selection.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-lts/008-selection.png rename to docs/data_management/transfer/images/gi-transfer-collection-to-lts/008-selection.png diff --git a/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-lts/009-transfer-started.png b/docs/data_management/transfer/images/gi-transfer-collection-to-lts/009-transfer-started.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-lts/009-transfer-started.png rename to docs/data_management/transfer/images/gi-transfer-collection-to-lts/009-transfer-started.png diff --git a/docs/data_management/transfer/images/gi-transfer-lts-to-cheaha/005-ready-to-select.png b/docs/data_management/transfer/images/gi-transfer-lts-to-cheaha/005-ready-to-select.png new file mode 100644 index 000000000..e02859bbf Binary files /dev/null and b/docs/data_management/transfer/images/gi-transfer-lts-to-cheaha/005-ready-to-select.png differ diff --git a/docs/data_management/transfer/images/gi-transfer-lts-to-cheaha/008-selection.png b/docs/data_management/transfer/images/gi-transfer-lts-to-cheaha/008-selection.png new file mode 100644 index 000000000..6042bb794 Binary files /dev/null and b/docs/data_management/transfer/images/gi-transfer-lts-to-cheaha/008-selection.png differ diff --git a/docs/data_management/transfer/images/gi-transfer-lts-to-cheaha/009-transfer-started.png b/docs/data_management/transfer/images/gi-transfer-lts-to-cheaha/009-transfer-started.png new file mode 100644 index 000000000..f0abfac41 Binary files /dev/null and b/docs/data_management/transfer/images/gi-transfer-lts-to-cheaha/009-transfer-started.png differ diff --git a/docs/data_management/transfer/images/gi-uab-collections/001-cheaha.png b/docs/data_management/transfer/images/gi-uab-collections/001-cheaha.png new file mode 100644 index 000000000..fedc07520 Binary files /dev/null and b/docs/data_management/transfer/images/gi-uab-collections/001-cheaha.png differ diff --git a/docs/data_management/transfer/tutorial/images/gi-uab-collections/002-lts.png b/docs/data_management/transfer/images/gi-uab-collections/002-lts.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-uab-collections/002-lts.png rename to docs/data_management/transfer/images/gi-uab-collections/002-lts.png diff --git a/docs/data_management/transfer/tutorial/images/gi-uab-collections/003-box.png b/docs/data_management/transfer/images/gi-uab-collections/003-box.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-uab-collections/003-box.png rename to docs/data_management/transfer/images/gi-uab-collections/003-box.png diff --git a/docs/data_management/transfer/images/gi-web-app/001-login.png b/docs/data_management/transfer/images/gi-web-app/001-login.png new file mode 100644 index 000000000..5fd7747fc Binary files /dev/null and b/docs/data_management/transfer/images/gi-web-app/001-login.png differ diff --git a/docs/data_management/transfer/images/gi-web-app/002-login.png b/docs/data_management/transfer/images/gi-web-app/002-login.png new file mode 100644 index 000000000..9b46af655 Binary files /dev/null and b/docs/data_management/transfer/images/gi-web-app/002-login.png differ diff --git a/docs/data_management/transfer/images/gi-web-app/002-search.png b/docs/data_management/transfer/images/gi-web-app/002-search.png new file mode 100644 index 000000000..41ecf9d34 Binary files /dev/null and b/docs/data_management/transfer/images/gi-web-app/002-search.png differ diff --git a/docs/data_management/transfer/tutorial/images/gi-web-app/003-select.png b/docs/data_management/transfer/images/gi-web-app/003-select.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-web-app/003-select.png rename to docs/data_management/transfer/images/gi-web-app/003-select.png diff --git a/docs/data_management/transfer/tutorial/images/gi-web-app/004-sso.png b/docs/data_management/transfer/images/gi-web-app/004-sso.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-web-app/004-sso.png rename to docs/data_management/transfer/images/gi-web-app/004-sso.png diff --git a/docs/data_management/transfer/tutorial/images/gi-web-app/005-file-manager-single-panel.png b/docs/data_management/transfer/images/gi-web-app/005-file-manager-single-panel.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-web-app/005-file-manager-single-panel.png rename to docs/data_management/transfer/images/gi-web-app/005-file-manager-single-panel.png diff --git a/docs/data_management/transfer/tutorial/images/gi-web-app/006-view-buttons.png b/docs/data_management/transfer/images/gi-web-app/006-view-buttons.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-web-app/006-view-buttons.png rename to docs/data_management/transfer/images/gi-web-app/006-view-buttons.png diff --git a/docs/data_management/transfer/tutorial/images/gi-web-app/007-file-manager-two-panel.png b/docs/data_management/transfer/images/gi-web-app/007-file-manager-two-panel.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-web-app/007-file-manager-two-panel.png rename to docs/data_management/transfer/images/gi-web-app/007-file-manager-two-panel.png diff --git a/docs/data_management/transfer/tutorial/images/gi-web-app/008-file-manager-two-panel-example.png b/docs/data_management/transfer/images/gi-web-app/008-file-manager-two-panel-example.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/gi-web-app/008-file-manager-two-panel-example.png rename to docs/data_management/transfer/images/gi-web-app/008-file-manager-two-panel-example.png diff --git a/docs/data_management/transfer/images/globus_100_shared_search_collections.png b/docs/data_management/transfer/images/globus_100_shared_search_collections.png index 40e5a7796..029df273e 100644 Binary files a/docs/data_management/transfer/images/globus_100_shared_search_collections.png and b/docs/data_management/transfer/images/globus_100_shared_search_collections.png differ diff --git a/docs/data_management/transfer/images/globus_101_shared_collections.png b/docs/data_management/transfer/images/globus_101_shared_collections.png index e343ba5dd..74c5f2eaf 100644 Binary files a/docs/data_management/transfer/images/globus_101_shared_collections.png and b/docs/data_management/transfer/images/globus_101_shared_collections.png differ diff --git a/docs/data_management/transfer/tutorial/images/go-choose-folder/mac/001-notification-area.png b/docs/data_management/transfer/images/go-choose-folder/mac/001-notification-area.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-choose-folder/mac/001-notification-area.png rename to docs/data_management/transfer/images/go-choose-folder/mac/001-notification-area.png diff --git a/docs/data_management/transfer/tutorial/images/go-choose-folder/mac/002-context-menu.png b/docs/data_management/transfer/images/go-choose-folder/mac/002-context-menu.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-choose-folder/mac/002-context-menu.png rename to docs/data_management/transfer/images/go-choose-folder/mac/002-context-menu.png diff --git a/docs/data_management/transfer/tutorial/images/go-choose-folder/mac/003-access-tab.png b/docs/data_management/transfer/images/go-choose-folder/mac/003-access-tab.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-choose-folder/mac/003-access-tab.png rename to docs/data_management/transfer/images/go-choose-folder/mac/003-access-tab.png diff --git a/docs/data_management/transfer/tutorial/images/go-choose-folder/win/001-sys-tray.png b/docs/data_management/transfer/images/go-choose-folder/win/001-sys-tray.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-choose-folder/win/001-sys-tray.png rename to docs/data_management/transfer/images/go-choose-folder/win/001-sys-tray.png diff --git a/docs/data_management/transfer/tutorial/images/go-choose-folder/win/002-context-menu.png b/docs/data_management/transfer/images/go-choose-folder/win/002-context-menu.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-choose-folder/win/002-context-menu.png rename to docs/data_management/transfer/images/go-choose-folder/win/002-context-menu.png diff --git a/docs/data_management/transfer/tutorial/images/go-choose-folder/win/003-access-tab-default.png b/docs/data_management/transfer/images/go-choose-folder/win/003-access-tab-default.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-choose-folder/win/003-access-tab-default.png rename to docs/data_management/transfer/images/go-choose-folder/win/003-access-tab-default.png diff --git a/docs/data_management/transfer/tutorial/images/go-choose-folder/win/004-access-tab-changed.png b/docs/data_management/transfer/images/go-choose-folder/win/004-access-tab-changed.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-choose-folder/win/004-access-tab-changed.png rename to docs/data_management/transfer/images/go-choose-folder/win/004-access-tab-changed.png diff --git a/docs/data_management/transfer/tutorial/images/go-choose-folder/win/005-general-tab-default.png b/docs/data_management/transfer/images/go-choose-folder/win/005-general-tab-default.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-choose-folder/win/005-general-tab-default.png rename to docs/data_management/transfer/images/go-choose-folder/win/005-general-tab-default.png diff --git a/docs/data_management/transfer/tutorial/images/go-choose-folder/win/006-general-tab-changed.png b/docs/data_management/transfer/images/go-choose-folder/win/006-general-tab-changed.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-choose-folder/win/006-general-tab-changed.png rename to docs/data_management/transfer/images/go-choose-folder/win/006-general-tab-changed.png diff --git a/docs/data_management/transfer/tutorial/images/go-create-collection/001-collection-tab.png b/docs/data_management/transfer/images/go-create-collection/001-collection-tab.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-create-collection/001-collection-tab.png rename to docs/data_management/transfer/images/go-create-collection/001-collection-tab.png diff --git a/docs/data_management/transfer/tutorial/images/go-create-collection/002-add-guest-collection.png b/docs/data_management/transfer/images/go-create-collection/002-add-guest-collection.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-create-collection/002-add-guest-collection.png rename to docs/data_management/transfer/images/go-create-collection/002-add-guest-collection.png diff --git a/docs/data_management/transfer/tutorial/images/go-create-collection/003-form.png b/docs/data_management/transfer/images/go-create-collection/003-form.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-create-collection/003-form.png rename to docs/data_management/transfer/images/go-create-collection/003-form.png diff --git a/docs/data_management/transfer/tutorial/images/go-create-collection/004-completion.png b/docs/data_management/transfer/images/go-create-collection/004-completion.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-create-collection/004-completion.png rename to docs/data_management/transfer/images/go-create-collection/004-completion.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/common/collection-details.png b/docs/data_management/transfer/images/go-gcp-install/common/collection-details.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/common/collection-details.png rename to docs/data_management/transfer/images/go-gcp-install/common/collection-details.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/common/consents.png b/docs/data_management/transfer/images/go-gcp-install/common/consents.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/common/consents.png rename to docs/data_management/transfer/images/go-gcp-install/common/consents.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/mac/001-select.png b/docs/data_management/transfer/images/go-gcp-install/mac/001-select.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/mac/001-select.png rename to docs/data_management/transfer/images/go-gcp-install/mac/001-select.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/mac/002-instructions.png b/docs/data_management/transfer/images/go-gcp-install/mac/002-instructions.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/mac/002-instructions.png rename to docs/data_management/transfer/images/go-gcp-install/mac/002-instructions.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/mac/003-download.png b/docs/data_management/transfer/images/go-gcp-install/mac/003-download.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/mac/003-download.png rename to docs/data_management/transfer/images/go-gcp-install/mac/003-download.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/mac/004-install.png b/docs/data_management/transfer/images/go-gcp-install/mac/004-install.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/mac/004-install.png rename to docs/data_management/transfer/images/go-gcp-install/mac/004-install.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/mac/005-application.png b/docs/data_management/transfer/images/go-gcp-install/mac/005-application.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/mac/005-application.png rename to docs/data_management/transfer/images/go-gcp-install/mac/005-application.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/mac/006-authenticate.png b/docs/data_management/transfer/images/go-gcp-install/mac/006-authenticate.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/mac/006-authenticate.png rename to docs/data_management/transfer/images/go-gcp-install/mac/006-authenticate.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/mac/009-notification-area-icon.png b/docs/data_management/transfer/images/go-gcp-install/mac/009-notification-area-icon.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/mac/009-notification-area-icon.png rename to docs/data_management/transfer/images/go-gcp-install/mac/009-notification-area-icon.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/win/001-select.png b/docs/data_management/transfer/images/go-gcp-install/win/001-select.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/win/001-select.png rename to docs/data_management/transfer/images/go-gcp-install/win/001-select.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/win/002-instructions.png b/docs/data_management/transfer/images/go-gcp-install/win/002-instructions.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/win/002-instructions.png rename to docs/data_management/transfer/images/go-gcp-install/win/002-instructions.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/win/003-download.png b/docs/data_management/transfer/images/go-gcp-install/win/003-download.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/win/003-download.png rename to docs/data_management/transfer/images/go-gcp-install/win/003-download.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/win/004-select-path.png b/docs/data_management/transfer/images/go-gcp-install/win/004-select-path.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/win/004-select-path.png rename to docs/data_management/transfer/images/go-gcp-install/win/004-select-path.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/win/005-finish.png b/docs/data_management/transfer/images/go-gcp-install/win/005-finish.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/win/005-finish.png rename to docs/data_management/transfer/images/go-gcp-install/win/005-finish.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/win/006-authenticate.png b/docs/data_management/transfer/images/go-gcp-install/win/006-authenticate.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/win/006-authenticate.png rename to docs/data_management/transfer/images/go-gcp-install/win/006-authenticate.png diff --git a/docs/data_management/transfer/tutorial/images/go-gcp-install/win/009-system-tray-icon.png b/docs/data_management/transfer/images/go-gcp-install/win/009-system-tray-icon.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-gcp-install/win/009-system-tray-icon.png rename to docs/data_management/transfer/images/go-gcp-install/win/009-system-tray-icon.png diff --git a/docs/data_management/transfer/tutorial/images/go-my-collections/003-your-collections.png b/docs/data_management/transfer/images/go-my-collections/003-your-collections.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-my-collections/003-your-collections.png rename to docs/data_management/transfer/images/go-my-collections/003-your-collections.png diff --git a/docs/data_management/transfer/tutorial/images/go-my-collections/004-file-manager-selection-made.png b/docs/data_management/transfer/images/go-my-collections/004-file-manager-selection-made.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-my-collections/004-file-manager-selection-made.png rename to docs/data_management/transfer/images/go-my-collections/004-file-manager-selection-made.png diff --git a/docs/data_management/transfer/tutorial/images/go-my-collections/005-collection-details-page.png b/docs/data_management/transfer/images/go-my-collections/005-collection-details-page.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-my-collections/005-collection-details-page.png rename to docs/data_management/transfer/images/go-my-collections/005-collection-details-page.png diff --git a/docs/data_management/transfer/tutorial/images/go-share-collection/001-collection-details-overview.png b/docs/data_management/transfer/images/go-share-collection/001-collection-details-overview.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-share-collection/001-collection-details-overview.png rename to docs/data_management/transfer/images/go-share-collection/001-collection-details-overview.png diff --git a/docs/data_management/transfer/tutorial/images/go-share-collection/002-collection-details-permissions-before.png b/docs/data_management/transfer/images/go-share-collection/002-collection-details-permissions-before.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-share-collection/002-collection-details-permissions-before.png rename to docs/data_management/transfer/images/go-share-collection/002-collection-details-permissions-before.png diff --git a/docs/data_management/transfer/tutorial/images/go-share-collection/003-add-permissions-form.png b/docs/data_management/transfer/images/go-share-collection/003-add-permissions-form.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-share-collection/003-add-permissions-form.png rename to docs/data_management/transfer/images/go-share-collection/003-add-permissions-form.png diff --git a/docs/data_management/transfer/tutorial/images/go-share-collection/004-permissions-confirmation.png b/docs/data_management/transfer/images/go-share-collection/004-permissions-confirmation.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-share-collection/004-permissions-confirmation.png rename to docs/data_management/transfer/images/go-share-collection/004-permissions-confirmation.png diff --git a/docs/data_management/transfer/tutorial/images/go-share-collection/005-colection-details-permissions-after.png b/docs/data_management/transfer/images/go-share-collection/005-colection-details-permissions-after.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-share-collection/005-colection-details-permissions-after.png rename to docs/data_management/transfer/images/go-share-collection/005-colection-details-permissions-after.png diff --git a/docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/001-groups-page.png b/docs/data_management/transfer/images/go-uab-ha-group-access/001-groups-page.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/001-groups-page.png rename to docs/data_management/transfer/images/go-uab-ha-group-access/001-groups-page.png diff --git a/docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/002-groups-search.png b/docs/data_management/transfer/images/go-uab-ha-group-access/002-groups-search.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/002-groups-search.png rename to docs/data_management/transfer/images/go-uab-ha-group-access/002-groups-search.png diff --git a/docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/003-uab-ha-overview.png b/docs/data_management/transfer/images/go-uab-ha-group-access/003-uab-ha-overview.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/003-uab-ha-overview.png rename to docs/data_management/transfer/images/go-uab-ha-group-access/003-uab-ha-overview.png diff --git a/docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/004-join-form.png b/docs/data_management/transfer/images/go-uab-ha-group-access/004-join-form.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/004-join-form.png rename to docs/data_management/transfer/images/go-uab-ha-group-access/004-join-form.png diff --git a/docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/005-pending.png b/docs/data_management/transfer/images/go-uab-ha-group-access/005-pending.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/005-pending.png rename to docs/data_management/transfer/images/go-uab-ha-group-access/005-pending.png diff --git a/docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/006-my-groups.png.png b/docs/data_management/transfer/images/go-uab-ha-group-access/006-my-groups.png.png similarity index 100% rename from docs/data_management/transfer/tutorial/images/go-uab-ha-group-access/006-my-groups.png.png rename to docs/data_management/transfer/images/go-uab-ha-group-access/006-my-groups.png.png diff --git a/docs/data_management/transfer/index.md b/docs/data_management/transfer/index.md index 9f748cc84..d5f71a653 100644 --- a/docs/data_management/transfer/index.md +++ b/docs/data_management/transfer/index.md @@ -2,14 +2,10 @@ We currently support the following tools on Cheaha to manage data transfers. You can find information for the following tools and services we support; -1. [Globus](./globus.md) +1. [Globus](./globus/index.md) 1. [Filezilla](./filezilla.md) 1. [Rclone](./rclone.md) -We also have tutorials to guide your data transfer needs; - -1. [Data Transfer (Globus)](../transfer/tutorial/index.md) - ## What Data Sharing Method Works Best for My Collaboration? Choosing the right data sharing tool/method depends on how often you transfer data and who needs access to it. Below are some common scenarios and our recommendations. @@ -21,12 +17,12 @@ If you are sending data infrequently, for example, to share results with a colla - If the transfer is a copy: You are keeping your own copy. Keep in mind that your version and the recipient’s version may be out of sync over time. - If the transfer is a move: After confirming the recipient has retrieved the data, it is good practice to delete your local copy to free up storage space. -- **Recommended method:** [Globus](../transfer/globus.md) is the easiest and most robust option. It is also simple to restrict who has access. +- **Recommended method:** [Globus](../transfer/globus/index.md) is the easiest and most robust option. It is also simple to restrict who has access. - Place your data in a directory you want to share. - - [Create a shared Globus collection](../transfer/globus.md#creating-a-guest-collection). - - [Share the collection](../transfer/tutorial/globus_organization_tutorial.md#how-do-i-share-a-collection-with-others) to the intended recipient. + - [Create a shared Globus collection](../transfer/globus/globus_organization_tutorial.md#creating-a-guest-collection). + - [Share the collection](../transfer/globus/globus_organization_tutorial.md#how-do-i-share-a-collection-with-others) to the intended recipient. -If you expect your collaboration to expand in the future, use a shared storage allocation ([GPFS](../cheaha_storage_gpfs/project_directories.md) or [LTS](../lts/index.md)). See our [Which Platform I Need](../../data_management/index.md#which-platform-do-i-need-an-allocation-for) page to choose the shared storage allocation that better fits your needs. This option allows ongoing collaboration but requires more attention to access control, especially if your data falls under security requirements such as HIPAA, NIST 800-171, or ITAR. Collaborations using shared allocations can be managed in two ways. +If you expect your collaboration to expand in the future, use a shared storage allocation ([GPFS](../../data_management/storage/cheaha_storage_gpfs/project_directories.md) or [LTS](../../data_management/storage/lts/index.md)). See our [Which Platform I Need](../../data_management/storage/index.md#which-platform-do-i-need-an-allocation-for) page to choose the shared storage allocation that better fits your needs. This option allows ongoing collaboration but requires more attention to access control, especially if your data falls under security requirements such as HIPAA, NIST 800-171, or ITAR. Collaborations using shared allocations can be managed in two ways. - Have the intended recipient add you to their shared allocation so you can retrieve/store data. - Add the intended recipient to your shared allocation so they can retrieve/store data. If the recipient is external, they will need to create a [XIAS account](../../account_management/xias/index.md). @@ -34,13 +30,13 @@ If you expect your collaboration to expand in the future, use a shared storage a Researchers are responsible for controlling access to data in their allocations. Each storage platform uses distinct access control methods. - GPFS: Set [ACLs](../../workflow_solutions/shell.md#manage-researcher-access-to-files-and-directories-getfacl-setfacl) on all other directories with restricted user permissions. -- LTS: Append a [bucket policy](../lts/iam_and_policies.md#bucket-policies) restricted to the specific data and specific user. +- LTS: Append a [bucket policy](../storage/lts/iam_and_policies.md#bucket-policies) restricted to the specific data and specific user. ## Frequent Transfer of Your Data -If you frequently transfer data, the same principles for infrequent data transfer still apply, but automation becomes important. Consider using Globus Flows or Globus Compute for automated and repeatable workflows. When multiple collaborators need to analyze the same data simultaneously, Globus may no longer be the best option. In that case, use a shared allocation ([GPFS](../cheaha_storage_gpfs/project_directories.md) or [LTS](../lts/index.md)) so everyone can work directly with the data in place. +If you frequently transfer data, the same principles for infrequent data transfer still apply, but automation becomes important. Consider using Globus Flows or Globus Compute for automated and repeatable workflows. When multiple collaborators need to analyze the same data simultaneously, Globus may no longer be the best option. In that case, use a shared allocation ([GPFS](../storage/cheaha_storage_gpfs/project_directories.md) or [LTS](../../data_management/storage/lts/index.md)) so everyone can work directly with the data in place. -If the collaborator is external, you can still transfer data regularly within shared allocations ([GPFS](../cheaha_storage_gpfs/project_directories.md) or [LTS](../lts/index.md)). However, they will need to create a [XIAS account](../../account_management/xias/index.md). +If the collaborator is external, you can still transfer data regularly within shared allocations ([GPFS](../storage/cheaha_storage_gpfs/project_directories.md) or [LTS](../../data_management//storage/lts/index.md)). However, they will need to create a [XIAS account](../../account_management/xias/index.md). !!! important diff --git a/docs/data_management/transfer/rclone.md b/docs/data_management/transfer/rclone.md index d2938e9e8..bb4b32edd 100644 --- a/docs/data_management/transfer/rclone.md +++ b/docs/data_management/transfer/rclone.md @@ -1,6 +1,6 @@ # RClone -[RClone](https://rclone.org/) is a powerful command line tool for transferring and synchronizing files over the internet between various machines, servers and cloud storage services. It is highly recommended for small to moderate amounts of data. For very large amounts of data consider using [Globus](globus.md) for increased robustness against failure. Where Globus is not available, `rclone` is still suitable. +[RClone](https://rclone.org/) is a powerful command line tool for transferring and synchronizing files over the internet between various machines, servers and cloud storage services. It is highly recommended for small to moderate amounts of data. For very large amounts of data consider using [Globus](globus/index.md) for increased robustness against failure. Where Globus is not available, `rclone` is still suitable. RClone requires a modest amount of setup time on local machines, but once setup can be used fairly easily. RClone uses the concepts of "remotes", which is an abstract term for any storage service or device that is not physically part of the local machine. Many remotes are offered, including [SFTP](../../uab_cloud/remote_access.md#sftp) and various [UAB Cloud Storage Solutions](https://www.uab.edu/it/home/tech-solutions/file-storage/storage-options). SFTP may be used to access Cheaha, cloud.rc and other laptop and desktop computers. diff --git a/docs/data_management/transfer/tutorial/globus_individual_tutorial.md b/docs/data_management/transfer/tutorial/globus_individual_tutorial.md deleted file mode 100644 index bcc7f2a86..000000000 --- a/docs/data_management/transfer/tutorial/globus_individual_tutorial.md +++ /dev/null @@ -1,367 +0,0 @@ - -# Globus Tutorials for Individual Researchers - -Do you have data and need to move it to a different computer system? Want to use a graphical interface to do it? Want a service that will attempt to resume interrupted transfers? Want enforced encryption for your transfers? Then Globus is right for you. - -If you are new to Globus, you are in the right place. If you represent a group looking to share data with others we recommend following these tutorials with our [Globus Tutorials for Research Groups](./globus_organization_tutorial.md). - -These tutorials are intended for individual researchers who need to move data from one location to another. If you manage a Research Core or lab and have never used Globus before, you'll want to start on this page. Then you will want to proceed to [Globus Tutorials for Research Groups](./globus_organization_tutorial.md). - -The tutorials below will teach you how to effectively use Globus for managing and transferring research data. We will explore what Globus is, why you might use it, how it works, and we'll walk you through the essential steps to get started. You will learn how to set up your Globus account, access the Globus application, find Collections shared with you, and search for Collections by name. - -Topics covered: - -1. [Why Globus?](#why-globus) -1. [How Does Globus Work?](#how-does-globus-work) -1. [Tutorial Prerequisites](#tutorial-prerequisites) -1. [How Do I Get Onto the Globus Web App?](#how-do-i-get-onto-the-globus-web-app) -1. [How Do I Search for Collections by Name?](#how-do-i-search-for-collections-by-name) -1. [How Do I Find UAB Storage Collections?](#how-do-i-find-uab-storage-collections) -1. [How Do I Find Collections Shared with Me?](#how-do-i-find-collections-shared-with-me) -1. [How Do I Transfer between a Collection and Cheaha?](#how-do-i-transfer-between-a-collection-and-cheaha) -1. [How Do I Transfer between a Collection and LTS?](#how-do-i-transfer-between-a-collection-and-lts) -1. [How Do I Transfer between LTS and Cheaha?](#how-do-i-transfer-between-lts-and-cheaha) -1. [How Do I Check Transfer Status?](#how-do-i-check-transfer-status) - -## Why Globus? - -Globus is a data transfer ecosystem that simplifies the process of transferring, sharing, and managing large datasets. It is used by research institutions around the world to move research data between different storage devices, computer systems, and institutions. Globus has many research data oriented features, making it ideal for many research data transfer scenarios. Below is a list of features. - -- Straight-forward, browser-based, graphical interface. -- Compatible with [UAB Box](https://www.uab.edu/it/home/tech-solutions/file-storage/box). -- Compatible with [Long-Term Storage](../../lts/index.md). -- Can be used with your laptop, desktop, or lab workstation via Globus Connect Personal (GCP). -- Transfers are automatically retried in the event of network or computer system outages. -- Transfers are encrypted end-to-end. Globus never sees your data. -- Suitable for transferring PHI and HIPAA data. Note: a UAB Enterprise IT risk assessment is required. - -## How Does Globus Work? - -Globus is an ecosystem of software intended to make research data transfer simpler. The Globus Web Application (Web App) at allows you to initiate transfers between any two Collections you have authorization to access. The Globus Connect Personal (GCP) software lets you turn any computer into a Globus Collection. At no point do Globus servers touch your research data. Instead, when you initiate a transfer between two Collections, the Globus application tells the two Collections that they need to talk to each other and data is sent directly between them. The Collections update the application with information you may need to know, such as how much data has transferred so far, how fast the transfer is proceeding, and any errors that occur. If the connection between Collections is interrupted for any reason, the Globus application will attempt to restart the transfer from where it left off. - -## Tutorial Prerequisites - -For these tutorials, you will need your BlazerID or [XIAS ID](../../../account_management/xias/index.md) and password to authenticate using UAB Single Sign-On (SSO). - -## How Do I Get Onto the Globus Web App? - -1. Use your browser to navigate to . You should see a login page like below. - - ![Globus Web App login page.](./images/gi-web-app/001-login.png) - -1. To login, first you must find and select our institution. Type "UAB" or "University of Alabama at Birmingham" into the search bar to locate UAB in the list. The image below shows the correct choice in a red box. - - - !!! note - - If you are an external collaborator using a [XIAS account](../../../account_management/xias/index.md) (this is uncommon) to interact with UAB-owned storage you will still need to search for "UAB". Do not use your home institution login to access UAB storage systems, as you will only have access to UAB storage with your XIAS credentials. - - - ![Globus Web App login search bar with UAB entered and University of Alabama at Birmingham in a red box.](./images/gi-web-app/002-search.png) - -1. Select "University of Alabama at Birmingham" from the drop-down menu. - - ![Globus Web App with University of Alabama at Birmingham selected showing activated Continue button.](./images/gi-web-app/003-select.png) - -1. Click "Continue" to be taken to the UAB Single Sign-On (SSO) form. Enter your BlazerID and password in the SSO form, then click "Log In". Complete the login process as usual. - - ![UAB Single Sign-On form.](./images/gi-web-app/004-sso.png) - -1. You should be taken to the File Manager page of the Globus Web App. We will be revisiting this page frequently throughout the tutorials. We highly recommend taking some time to familiarize yourself with the page as you proceed. The next few steps outline the important features of the File Manager page and its purpose. - - ![Globus Web App file manager single-panel view.](./images/gi-web-app/005-file-manager-single-panel.png) - -1. Recall that Globus is a data transfer application. A data transfer means moving data between two computers: source and destination. As we learn more about Globus, data stored on the source and destination will be visible on the File Manager page in the panels in the screenshot above. - - By default, only one panel is visible. We recommend selecting two-panel view mode for improved ease of use. Select two-panel view mode by clicking the button located near the top right corner, as shown in the red box below. Of the three available view mode buttons, the two-panel view mode button is in the center. - - From here on, tutorials will assume you are using two-panel view mode when we refer to the File Manager page. - - ![Globus Web App file manager two-panel view.](./images/gi-web-app/006-view-buttons.png) - -1. Now that two-panel view mode has been selected, you will see two sets of features and panels side-by-side. Most of these features will be used extensively when using Globus. - - - **(1) Collection Search bar**: Clicking here will open the Collection Search page, allowing you to find Collections of data shared by others, or find your own Collections. The next step will demonstrate features of the Collection Search page. - - **(2) Path text field**: After a Collection has been selected using (1), this text field allows you to enter the path to a specific folder, updating the files and folders shown in (5). - - **(3) Start Transfer button**: When you have selected files and folders in (5), use this button to start transferring. - - **(4) Transfer Options drop down menu**: Various options for the transfer are available in this menu. These options are not commonly needed and are not used in the tutorials. - - **(5) File Browser panel**: After a Collection has been selected in (1), a list of files and folders will appear here for the path shown in (2). You may navigate the Collection here like you would on your operating system. - - **(6) Navigation panel**: The blue bar at the left hand side of the Globus Web App is the navigation panel. It will be referred to and used many times throughout these tutorials. - - ![Globus Web App file manager two-panel view.](./images/gi-web-app/007-file-manager-two-panel.png) - -1. To better prepare you for what to expect, the screenshot below shows the File Manager page with an example Collection selected, called "my-gcp-collection". This example shows a path in the Path text field and contents in the File Browser panel. - - - **(1) Select All checkbox**: Select or unselect all files in the currently displayed folder. - - **(2) Go Up One Level button**: Go to the folder containing the currently displayed folder. - - **(3) Refresh button**: Refreshes the view of the currently displayed folder. - - **(4) File Selection checkbox**: If checked, then the file is selected for transfer. This file is selected. - - **(5) Folder Selection checkbox**: If checked, then the folder is currently selected for transfer. This folder is not selected. - - ![Globus Web App file manager two-panel view with Collection selected showing a file and folder.](./images/gi-web-app/008-file-manager-two-panel-example.png) - -The File Manager page will be your most frequently-visited page when using Globus for data transfers. It is central to usage of the Globus Web Application. Please take some time to familiarize yourself with its look and feel. As you progress in the tutorials, please take time to experiment with transferring data to better understand how the interface works. Feel free to return here for guidance. - -From here you can proceed to [How Do I Search for Collections by Name?](#how-do-i-search-for-collections-by-name) - -[Return to the top](#globus-tutorials-for-individual-researchers). - -## How Do I Search for Collections by Name? - -Please follow these instructions to search for a specific Collection by name. - -1. [Get onto the Globus Web App](#how-do-i-get-onto-the-globus-web-app). -1. You should see the File Manager page. If not click File Manager in the left-hand navigation panel. - - ![File Manager page.](./images/common/file-manager.png) - -1. Click the Collection Search bar to open the Collection Search page. - - ![Collection search page](./images/common/collection-search-page.png) - -1. In the example below, we typed "uab box" to search for the UAB Box Collection, which connects to UAB's Box.com service. - - ![Globus Collection search page showing results of searching for "UAB Box".](./images/gi-search-collections/003-results.png) - -1. Click the name of the Collection to be taken back to the file manager page with the Collection filled in. - - ![File Manager page of the Globus Web App. One of the Collection Search bars is filled in with the previously selected Collection.](./images/gi-find-shared-collections/004-selected.png) - -If you can't find a particular Collection this way, but know it was shared with you, try [finding Collections shared with me](#how-do-i-find-collections-shared-with-me). Proceed on to learn [how to find UAB storage Collections](#how-do-i-find-uab-storage-collections). - -[Return to the top](#globus-tutorials-for-individual-researchers). - -## How Do I Find UAB Storage Collections? - -UAB offers [multiple storage resources](../../index.md). The following resources may be accessed through Globus Collections. - -- Cheaha file system (GPFS) including [individual directories](../../index.md#what-individual-storage-solutions-are-available) and [project directories](../../cheaha_storage_gpfs/project_directories.md). -- [Long-Term Storage (LTS)](../../lts/index.md). -- [UAB Box](https://www.uab.edu/it/home/tech-solutions/file-storage/box). - -To find these resources on Globus, use the following steps. - -1. [Get onto the Globus Web App](#how-do-i-get-onto-the-globus-web-app). -1. You should see the File Manager page. If not click File Manager in the left-hand navigation panel. - - ![File Manager page.](./images/common/file-manager.png) - -1. Click the Collection Search bar to open the Collection Search page. - - ![Collection search page](./images/common/collection-search-page.png) - -1. In the search bar, type one of the following, depending on which resource you need, and select the appropriate entry. - - - **Cheaha Filesystem (GPFS)**: Type "UAB Cheaha". There are two Collections, choose one based on where the other computer is located. - - **(1)** Transferring with a computer on the UAB Campus Network or UAB Wifi? Select "Cheaha cluster **on**-campus (UAB Science DMZ)". - - **(2)** Transferring with a computer on other networks? select "Cheaha cluster **off**-campus (UAB Science DMZ)". - - ![UAB Box search results.](./images/gi-uab-collections/001-cheaha.png) - - - **Long-Term Storage (LTS)**: Type "UAB LTS" and select the entry labeled "UAB Research Computing LTS (Long Term Storage aka S3)". - - ![UAB Box search results.](./images/gi-uab-collections/002-lts.png) - - - **UAB Box**: Type "UAB Box" and select the entry labeled "UAB Box". - - ![UAB Box search results.](./images/gi-uab-collections/003-box.png) - -Proceed on to learn [how to find Collections shared with you](#how-do-i-find-uab-storage-collections). - -[Return to the top](#globus-tutorials-for-individual-researchers). - -## How Do I Find Collections Shared With Me? - -Globus allows users to share Collections with others. Other researchers, labs, and Research Cores on campus may invite you to their Collections to share data with you. The following instructions let you view Collections shared with you. - -1. [Log in to Globus using your credentials](#how-do-i-get-onto-the-globus-web-app) -1. You should see the File Manager page. If not click File Manager in the left-hand navigation panel. - - ![File Manager page.](./images/common/file-manager.png) - -1. Click the Collection Search bar to open the Collection Search page. - - ![Collection search page](./images/common/collection-search-page.png) - -1. Click the Shared With You tab. The list of Collections will be filtered down to all Collections others have granted you access to, which should help you find the Collections you need. - - ![Collection Search page of Globus Web App. Shared With You tab is selected and the list of Collections has been filtered down.](./images/gi-find-shared-collections/003-list.png) - -1. Click the name of the Collection to be taken back to the file manager page with the Collection filled in. - - ![File Manager page of the Globus Web App. One of the Collection Search bars is filled in with the previously selected Collection.](./images/gi-find-shared-collections/004-selected.png) - -If you can't find a particular Collection this way, but know its name, try [searching for Collections](#how-do-i-search-for-collections-by-name). - -Proceed on to learn how to transfer between Collections. - -- [Between a Collection and Cheaha](#how-do-i-transfer-between-a-collection-and-cheaha) -- [Between a Collection and LTS](#how-do-i-transfer-between-a-collection-and-lts) -- [Between LTS and Cheaha](#how-do-i-transfer-between-lts-and-cheaha) - -[Return to the top](#globus-tutorials-for-individual-researchers). - -## How Do I Transfer Between a Collection and Cheaha? - -Please use the following instructions to transfer data from a Collection to [Cheaha GPFS](../../index.md#what-type-of-storage-do-i-need). These instructions may also be used for other buckets on LTS, provided you have access and know their file path. - -The direction of transfer may also be reversed to transfer data from Cheaha to a Collection. Note that some Collections may not allow you to transfer data back, such as some of those provided by Research Cores. - -These instructions can be generalized to any two Collections you have access to on Globus. - -1. [Log in to Globus using your credentials](#how-do-i-get-onto-the-globus-web-app). - -1. Find the Collection [by filtering](#how-do-i-find-collections-shared-with-me) or [by searching](#how-do-i-search-for-collections-by-name) in the left side Collection Search bar. - -1. Once you have selected the Collection you wish to transfer data from, repeat the process to [search](#how-do-i-search-for-collections-by-name) for a Cheaha Collection in the Collection Search bar on the right side of the file manager page. Search for "Cheaha cluster" to find them. - - Pay close attention in choosing which of the two you need. Choosing incorrectly could lead to slow transfers. Answer the following questions to help you decide. Is the first Collection you selected... - - - ...part of a lab or Research Core on campus? Select "on-campus". - - ...on a computer on the UAB Campus Network or the UAB Campus VPN? Select "on-campus". - - ...at a different institution? Select "off-campus". - - ...on a computer on a home network? Select "off-campus". - -1. When you select a Cheaha Collection, or any other High Assurance (HA) Collection or Guest Collection, you will be prompted to re-authenticate. Click the Continue button to do so, then select your UAB email address. - - ![High Assurance request for reauthentication.](./images/common/ha-authenticate.png) - - ![Identity selection showing a UAB email address.](./images/common/select-identity.png) - -1. At this point, your file manager page should look something like the following image. At this point, both Collection Search bars should have a Collection filled in. The left side should be the Collection you wish to transfer from. The right side should be a Cheaha Collection. You should see files and folders on both sides. - - ![File Manager page showing a Collection and Cheaha Collection.](./images/gi-transfer-collection-to-cheaha/005-ready-to-select.png) - -1. Locate the path on the source Collection side. Either type the path into the Path field manually, or use the graphical selection field to click on folder names to navigate the filesystem. - -1. Repeat the process on the Cheaha Collection side to locate the destination path on the Cheaha Collection side. - -1. Select the file and folders you wish to transfer on the source Collection side. Do so by clicking the checkboxes next to the file and folder names. - - ![File Manager page with a file selected in the left selection area. The start button is boxed in red.](./images/gi-transfer-collection-to-cheaha/008-selection.png) - -1. To start the transfer, click the "Start" button on the side you made your selections. A transfer will be started and you should see a green toast notification at the upper-right corner of the web page. Press the "X" button to dismiss the notification or click "View Details" to be taken to the Activity page to see more details about the transfer. - - ![File Manager page with a toast notification indicating the file transfer started successfully.](./images/gi-transfer-collection-to-cheaha/009-transfer-started.png) - -From here you can proceed to other related tutorials to initiate other transfers or -[return to the index](#globus-tutorials-for-individual-researchers). - -- [How Do I check transfer status?](#how-do-i-check-transfer-status) -- [How Do I Transfer Between a Collection and LTS?](#how-do-i-transfer-between-a-collection-and-lts) -- [How Do I Transfer Between LTS and Cheaha?](#how-do-i-transfer-between-lts-and-cheaha) - -[Return to the top](#globus-tutorials-for-individual-researchers). - -## How Do I Transfer Between a Collection and LTS? - -Please use the following instructions to transfer data from a Collection to a [bucket on LTS](../../lts/index.md). These instructions may also be used for other directories on Cheaha, provided you have access and know their file path. - -The direction of transfer may also be reversed to transfer data from LTS to the destination Collection. Note that some Collections may not allow you to transfer data back, such as some of those provided by Research Cores. - -These instructions can be generalized to any two Collections you have access to on Globus. - -1. [Log in to Globus using your credentials](#how-do-i-get-onto-the-globus-web-app). - -1. Find a Collection [by filtering](#how-do-i-find-collections-shared-with-me) or [by searching](#how-do-i-search-for-collections-by-name) in the left side Collection Search bar. - -1. Once you have selected the shared Collection you wish to transfer data from, repeat the process to [search](#how-do-i-search-for-collections-by-name) for the LTS Collection in the Collection Search bar on the right side of the file manager page. Search for "UAB LTS" to find it. - -1. When you select the LTS Collection, or an High Assurance (HA) Collection, you will be prompted to re-authenticate. Click the "Continue" button to do so, then select your UAB email address. - - ![High Assurance request for re-authentication.](./images/common/ha-authenticate.png) - - ![Identity selection showing a UAB email address.](./images/common/select-identity.png) - -1. At this point, your file manager page should look something like the following image. Both Collection Search bars should have a Collection filled in. The left side should be the Collection you wish to transfer from. The right side should be the "UAB Research Computing LTS (Long Term Storage aka S3)" Collection. You should see files and folders on both sides. - - ![File Manager page showing a Collection and LTS Collection.](./images/gi-transfer-collection-to-lts/005-ready-to-select.png) - -1. Locate the path on the source Collection side. Either type the path into the Path field manually, or use the graphical selection field to click on folder names to navigate the filesystem. - -1. Repeat the process on the LTS Collection side to locate the destination path on the LTS Collection side. - -1. Select the file and folders you wish to transfer on the source Collection side. Do so by clicking the checkboxes next to the file and folder names. - - ![File Manager page with a file selected in the left selection area. The start button is boxed in red.](./images/gi-transfer-collection-to-lts/008-selection.png) - -1. To start the transfer, click the "Start" button on the side you made your selections. A transfer will be started and you should see a green toast notification at the upper-right corner of the web page. Press the "X" button to dismiss the notification or click "View Details" to be taken to the Activity page to see more details about the transfer. - - ![File Manager page with a toast notification indicating the file transfer started successfully.](./images/gi-transfer-collection-to-lts/009-transfer-started.png) - -From here you can proceed to other related tutorials to initiate other transfers or -[return to the index](#globus-tutorials-for-individual-researchers). - -- [How do I check transfer status?](#how-do-i-check-transfer-status) -- [How Do I Transfer Between a Collection and Cheaha?](#how-do-i-transfer-between-a-collection-and-cheaha) -- [How Do I Transfer Between LTS and Cheaha?](#how-do-i-transfer-between-lts-and-cheaha) - -[Return to the top](#globus-tutorials-for-individual-researchers). - -## How Do I Transfer Between LTS and Cheaha? - -Please use the following instructions to transfer data from a [bucket on LTS](../../lts/index.md) to a project directory on Cheaha. These instructions may also be used for other buckets on LTS and directories on Cheaha, provided you have access and know their file paths. - -The direction of transfer may also be reversed to transfer data from Cheaha to LTS. - -These instructions can be generalized to any two Collections or Collections you have access to on Globus. - -1. [Log in to Globus using your credentials](#how-do-i-get-onto-the-globus-web-app). - -1. Find the LTS Collection by [searching](#how-do-i-search-for-collections-by-name) in the Collection Search bar on the left side of the file manager page. Search for "UAB LTS" to find it. - -1. Once you have selected the LTS Collection, repeat the process to [search](#how-do-i-search-for-collections-by-name) for a Cheaha Collection in the Collection Search bar on the right side of the file manager page. Search for "Cheaha cluster" to find it. - - Pay close attention in choosing which of the two you need. Choosing incorrectly could lead to slow transfers. Answer the following questions to help you decide. Is the first Collection you selected... - - - ...part of a lab or Research Core on campus? Select "on-campus". - - ...on a computer on the UAB Campus Network or UAB Wifi or the UAB Campus VPN? Select "on-campus". - - ...at a different institution? Select "off-campus". - - ...on a computer on a home network? Select "off-campus". - -1. Both the LTS and Cheaha Collections are High Assurance (HA) Collections and you will be prompted to re-authenticate. Click the "Continue" button to do so, then select your UAB email address. - - ![High Assurance request for reauthentication.](./images/common/ha-authenticate.png) - - ![Identity selection showing a UAB email address.](./images/common/select-identity.png) - -1. At this point, your file manager page should look something like the following image. Both Collection Search bars should have a Collection name filled in. The left side should be the LTS Collection. The right side should be a Cheaha Collection. You should see files and folders on both sides. - - ![File Manager page showing a shared Collection and LTS Collection.](./images/gi-transfer-lts-to-cheaha/005-ready-to-select.png) - -1. Locate the source path on the LTS Collection side. Either type the path into the Path field manually, or use the graphical selection field to click on folder names to navigate the filesystem. - -1. Repeat the process on the Cheaha Collection side to locate the destination path on the Cheaha Collection side. - -1. Select the file and folders you wish to transfer on the source Collection side. Do so by clicking the checkboxes next to the file and folder names. - - ![File Manager page with a file selected in the left selection area. The start button is boxed in red.](./images/gi-transfer-lts-to-cheaha/008-selection.png) - -1. To start the transfer, click the "Start" button on the side you made your selections. A transfer will be started and you should see a green toast notification at the upper-right corner of the web page. Press the "X" button to dismiss the notification or click "View Details" to be taken to the Activity page to see more details about the transfer. - - ![File Manager page with a toast notification indicating the file transfer started successfully.](./images/gi-transfer-lts-to-cheaha/009-transfer-started.png) - -From here you can proceed to other related tutorials to initiate other transfers or -[return to the index](#globus-tutorials-for-individual-researchers). - -- [How do I check transfer status?](#how-do-i-check-transfer-status) -- [How Do I Transfer Between a Collection and Cheaha?](#how-do-i-transfer-between-a-collection-and-cheaha) -- [How Do I Transfer Between a Collection and LTS?](#how-do-i-transfer-between-a-collection-and-lts) - -[Return to the top](#globus-tutorials-for-individual-researchers). - -## How Do I Check Transfer Status? - -To check the status of your transfers, please follow the instructions below. - -1. [Log in to Globus using your credentials](#how-do-i-get-onto-the-globus-web-app). - -1. In the Globus Web App, click "Activity" in the left navigation panel to go to the Activity page. There will be a list overview of transfers, with the most recent at the top. - - ![Globus Web App Activity page showing a successful transfer and failed transfer.](./images/gi-check-transfer-status/001-activity.png) - -1. To see more details about a transfer, click the transfer title. There will be two tabs. The Overview tab will have information and statistics about the transfer. The Event Log tab will have information about events that occurred during transfer, including start, stop, and any errors. The Event Log is useful for diagnosing issues with failed transfers. - - ![Overview page for unsuccessful transfer.](./images/gi-check-transfer-status/002-status.png) - -[Return to the top](#globus-tutorials-for-individual-researchers). diff --git a/docs/data_management/transfer/tutorial/globus_organization_tutorial.md b/docs/data_management/transfer/tutorial/globus_organization_tutorial.md deleted file mode 100644 index b290c62d3..000000000 --- a/docs/data_management/transfer/tutorial/globus_organization_tutorial.md +++ /dev/null @@ -1,411 +0,0 @@ -# Globus Tutorials for UAB Research Organizations - -This tutorial is designed to guide UAB research managers, such as Lab PIs, Core Directors, and their management staff, to help them set up and use Globus Connect Personal (GCP) for secure data sharing on their local/personal computers. GCP allows you to share data with collaborators efficiently meeting security and compliance standards for file sharing. - -If you are new to Globus, we recommend starting with our [Globus Tutorials for Individual Researchers](./globus_individual_tutorial.md) first to familiarize yourself with how Globus is used. When those are complete, we then recommend following the tutorials on this page in order from start to finish, as later tutorials assume the previous tutorials have been completed. - -Topics covered: - -1. [Why Globus?](#why-globus) -1. [How Does Globus Work?](#how-does-globus-work) -1. [Tutorial Prerequisites](#tutorial-prerequisites) -1. [How Do I Get onto the Globus Web App?](#how-do-i-get-onto-the-globus-web-app) -1. [How Do I Install Globus Connect Personal?](#how-do-i-install-globus-connect-personal) - - [Windows](#installing-gcp-on-windows) - - [MacOS](#installing-gcp-on-macos) -1. [How Do I Choose Specific Folders to Share Using Globus Connect Personal?](#how-do-i-choose-specific-folders-to-share-using-globus-connect-personal) - - [Windows](#choose-specific-folders-on-windows) - - [MacOS](#choose-specific-folders-on-macos) -1. [How Do I Find Collections I Created or Own?](#how-do-i-find-collections-i-created-or-own) -1. [How Do I Enable Sharing for My Globus Account?](#how-do-i-enable-collection-sharing-for-my-globus-account) -1. [How Do I Create a Collection?](#how-do-i-create-a-collection) -1. [How Do I Share a Collection with Others?](#how-do-i-share-a-collection-with-others) -1. [How Do I Share Data with a Research Core Customer?](#how-do-i-share-data-with-a-research-core-customer) - -## Why Globus? - -Globus is a data transfer ecosystem that simplifies the process of transferring, sharing, and managing large datasets. It is used by research institutions around the world to move research data between different storage devices, computer systems, and institutions. Globus has many research data oriented features, making it ideal for many research data transfer scenarios. Below is a list of features. - -- Straight-forward, browser-based interface. -- Compatible with [Long-Term Storage](../../lts/index.md). -- Can be used to share data with Research Core customers. -- Can be used to transfer data between lab workstations, servers, and Cheaha. -- Transfers are automatically retried in the event of network or computer system outages. -- Transfers are encrypted end-to-end. Globus never sees your data. -- Suitable for transferring PHI and HIPAA data. Note: a UAB Enterprise IT risk assessment is required. - -## How Does Globus Work? - -Globus is an ecosystem of software intended to make research data transfer simpler. The Globus Web Application (Web App) at allows you to initiate transfers between any two Collections you have authorization to access. The Globus Connect Personal (GCP) and Globus Connect Server (GCS) software let you turn any computer into a Globus Collection. At no point do Globus servers touch your research data. Instead, when you initiate a transfer between two Collections, the Globus application tells the two Collections that they need to talk to each other and data is sent directly between them. The Collections update the application with information you may need to know, such as how much data has transferred so far, how fast the transfer is proceeding, and any errors that occur. If the connection between Collections is interrupted for any reason, the Globus application will attempt to restart the transfer from where it left off. - -## Tutorial Prerequisites - -For these tutorials, you will need your BlazerID or [XIAS ID](../../../account_management/xias/index.md) and password to authenticate using UAB Single Sign-On (SSO). - -## How Do I Get Onto the Globus Web App? - -To learn how to get onto the Globus Web App [Globus Tutorials for Individual Researchers Page](./globus_individual_tutorial.md#how-do-i-get-onto-the-globus-web-app). Please visit that link and then return here when you have finished. - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -## How Do I Install Globus Connect Personal? - -Globus Connect Personal (GCP) is available to install on the following Operating Systems. Linux is also available, but we do not cover the installation process here. If you need to install GCP on Linux, please refer to the [official documentation](https://docs.globus.org/globus-connect-personal/install/linux/). - -Managing data for a Research Core? Please stop here. You should know that Globus Connect Personal is _not_ suitable for production use with customers. Please [Contact Support](../../../help/support.md#how-do-i-create-a-support-ticket) to discuss setting up Globus Connect Server for your Research Core. - -Representing a lab? Globus Connect Personal may be suitable for your use case. If you transfer data infrequently, or transfer small amounts of data, then GCP is probably sufficient. If you frequently transfer large amounts of data, then Globus Connect Server can enable higher data transfer rates through parallelization. Please [Contact Support](../../../help/support.md) if you want to discuss installation of Globus Connect Server. - -GCP installation instructions for: - -- [Windows](#installing-gcp-on-windows) -- [MacOS](#installing-gcp-on-macos) - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -### Installing GCP on Windows - -The following steps will guide you to install Globus Connect Personal (GCP) on your computer running Windows. - -1. Navigate to the [Globus Connect Personal official page](https://www.globus.org/globus-connect-personal) and scroll down to find the GCP Windows version. Click on the "INSTALL NOW" button in the red box, as shown in the image below, to be taken to the official installation instructions and download link for GCP for Windows. The "INSTALL NOW" button is located below the text "Globus Connect Personal for Windows". - - ![Globus Connect Personal official page showing operating system options with links. Windows is in the center of the three.](./images/go-gcp-install/win/001-select.png) - -1. Click the "Download Globus Connect Personal" link on the instructions page, as shown below. This will redirect you to the GCP for Windows installer. - - ![Globus Connect Personal for Windows official instructions page with link to download GCP for Windows.](./images/go-gcp-install/win/002-instructions.png) - -1. As shown in the image below, click the "Download Globus Connect Personal for Windows" button to download the installer to download the installer to your computer. - - ![Download GCP Installer for Windows](./images/go-gcp-install/win/003-download.png) - -1. Find the installer on your computer and open it. Select where you would prefer to have your GCP installed and click the "Install" button. - - ![GCP installer window showing selected path and install button.](./images/go-gcp-install/win/004-select-path.png) - - Please note you must have administrator permissions, to do this. If you are unable to do so, you will need to contact the IT department responsible for managing your computer. - -1. When the installation is complete, click the "Finish" button to complete the GCP installation. - - ![Completing GCP Setup highlighting Finish button.](./images/go-gcp-install/win/005-finish.png) - -1. Following installation, GCP will launch in a new window. If it does not, look for it in your Start Menu. - - When GCP has started, click the "Log In" button to authenticate with Globus to begin the Collection setup process. This is a one-time setup to configure GCP to allow your machine to act as a Collection, enabling research data transfer with your computer. - - ![GCP Setup window with Log In button.](./images/go-gcp-install/win/006-authenticate.png) - - Note that if you uninstall and reinstall GCP, you will need to complete this process again. You should not need to repeat this process otherwise. - -1. Grant the required consents. This is required to set up your computer as a Collection. Also provide a name for your consents. We recommend choosing a name that is short, memorable, and related to the purpose for the Collection. - - ![GCP Setup form with required consents and Collection name field.](./images/go-gcp-install/common/consents.png) - -1. Enter the details for your GCP Collection, and click save to continue. The following list describes the fields in the form shown below. - - - **Owner Identity:** is the person responsible for this Collection. This field should already be filled with UAB Campus or XIAS email address. If not, please select that email address here. - - **Collection Name:** is the name for the Collection. This should be filled with the name of the Collection from the previous step. - - **Description:** Feel free to enter descriptive information about the Collection here. This information will be displayed in the Globus Web App when the Collection is viewed by others. - - **High Assurance:** Only check this box if the Collection has or will have PHI, HIPAA, or other protected data. If this is the case, please ensure that you have already completed a risk assessment with UAB Enterprise IT. - - ![GCP Setup Collection details](./images/go-gcp-install/common/collection-details.png) - -1. GCP Setup is now complete on your computer. Your computer is now serving your new Globus Collection and may be used to transfer data. Click "Exit Setup" to close the window. - -1. After installation, locate your Windows System Tray. In it you should see a white lowercase letter "g" in a filled blue circle. If you do not, try finding the Globus Connect Personal application in your start menu and starting the application. - - ![GCP Icon in Windows system tray.](./images/go-gcp-install/win/009-system-tray-icon.png) - -By default your Documents folder (typically `C:/Users/%username%/Documents`) is listed in your new Globus Collection. To change it continue on with [How Do I Choose Specific Folders Using Globus Connect Personal?](#how-do-i-choose-specific-folders-to-share-using-globus-connect-personal) - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -### Installing GCP on MacOS - -The following steps will guide you to install Globus Connect Personal (GCP) on your computer running MacOS. - -1. Navigate to the [Globus Connect Personal official page](https://www.globus.org/globus-connect-personal) and scroll down to find the GCP MacOS version. Click on the "INSTALL NOW" button in the red box in the image below to be taken to the official installation instructions and download link for GCP for MacOS. The "INSTALL NOW" button is located below the text "Globus Connect Personal for Mac". - - ![Globus Connect Personal official page showing operating system options with links. MacOS is the left of the three.](./images/go-gcp-install/mac/001-select.png) - -1. Click the "Download Globus Connect Personal" link on the instructions page, as shown below. This will redirect you to the GCP for MacOS installer. - - ![Globus Connect Personal for MacOS official instructions page with link to download GCP for MacOS.](./images/go-gcp-install/mac/002-instructions.png) - -1. As shown in the image below, click the "Download Globus Connect Personal for MacOS" button to download the installer to download the installer to your computer. - - ![Download GCP Installer for Windows](./images/go-gcp-install/mac/003-download.png) - -1. Find the installer on your computer and open it. A new window will pop-up asking you to drag the Globus Connect Personal app into the Application folder on your computer. Do so to install GCP. - - ![Copy Mac GCP App into the Applications Folder](./images/go-gcp-install/mac/004-install.png) - -1. When the above step is completed navigate to your Application folder and look for the "Globus Connect Personal" application. Open it to proceed. - - ![GCP application in the Applications Folder](./images/go-gcp-install/mac/005-application.png) - -1. When GCP has started, click on "Log In" to authenticate with Globus to begin the Collection setup process. This is a one-time setup to configure GCP to allow your machine to act as a Collection, enabling research data transfer with your computer. - - ![GCP Setup window with Log In button.](./images/go-gcp-install/mac/006-authenticate.png) - - Note that if you uninstall and reinstall GCP, you will need to complete this process again. You should not need to repeat this process otherwise. - -1. Grant the required consents. This is required to set up your computer as a Collection. Also provide a name for your Collection. We recommend choosing a name that is short, memorable, and related to the purpose for the Collection. - - ![GCP Setup form with required consents and Collection name field.](./images/go-gcp-install/common/consents.png) - -1. Enter the details for your GCP Collection, and click save to continue. The following list describes the fields in the form shown below. - - - **Owner Identity:** is the person responsible for this Collection. This field should already be filled with UAB Campus or XIAS email address. If not, please select that email address here. - - **Collection Name:** is the name for the Collection. This should be filled with the name of the Collection from the previous step. - - **Description:** Feel free to enter descriptive information about the Collection here. This information will be displayed in the Globus Web App when the Collection is viewed by others. - - **High Assurance:** Only check this box if the Collection has or will have PHI, HIPAA, or other protected data. If this is the case, please ensure that you have already completed a risk assessment with UAB Enterprise IT. - - ![GCP Setup Collection details](./images/go-gcp-install/common/collection-details.png) - -1. GCP Setup is now complete on your computer. Your computer is now serving your new Globus Collection and may be used to transfer data. Click "Exit Setup" to close the window. - -1. After installation locate your MacOS notification, generally at the top-right of your display. In it you should see a white lowercase letter "g" in a fille black circle. - - ![GCP Icon in MacOS notification area.](./images/go-gcp-install/mac/009-notification-area-icon.png) - -By default the `/Users/` folder is listed in your new Globus Collection. To change it continue on with [How Do I Share Specific Folders Using Globus Connect Personal?](#how-do-i-choose-specific-folders-to-share-using-globus-connect-personal) - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -## How Do I Choose Specific Folders to Share Using Globus Connect Personal? - -Please follow the instructions in this section to share a folder on your computer with others through your [Globus Connect Personal (GCP) Collection](#how-do-i-install-globus-connect-personal). - -We have instructions for the following Operating Systems. - -- [Windows](#choose-specific-folders-on-windows) -- [MacOS](#choose-specific-folders-on-macos) - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -### Choose Specific Folders on Windows - -1. In your Windows system tray, locate the icon that looks like a small letter "g" in a circle. This is the icon for Globus Connect Personal. If you cannot locate the icon in the system tray, then open the Globus Connect Personal app on your computer and look for it again. - - ![Expanded system tray showing icon of a small letter "g" in a circle.](./images/go-choose-folder/win/001-sys-tray.png) - -1. Right-click the icon to open the context menu and click "Options...". - - ![Context menu of Globus system tray icon showing options.](./images/go-choose-folder/win/002-context-menu.png) - -1. A new window will appear with a tab labelled Access. In the Access tab is an interface to configure folders available on your GCP Collection. For most use cases, you should not check the writeable checkbox. Below is a summary of what each part of the menu does. - - - **(1) Accessible Folders** table with Folder, Shareable and Writeable columns. Any folder listed here will appear on your GCP Collection. Your research data folder or directories must be listed here to be shareable. - - **(2) Shareable** column checkboxes controlling which folders can be shared with other users. Each of your research data directories must have this checkbox ticked to be shareable from the Collection. **Check this box only if you want to share your data with others.** - - **(3) Writeable** column checkboxes controlling which folders can be written to by other users. If a folder is shared with other users, then they will be able to add, delete, or change the contents. We recommend against ticking these boxes for Research Cores serving data to customers. **Check this box only if you want others to be able to change your data.** - - **(4) Plus `+` and minus `-` buttons** that allow you to add or remove folders from the list. - - **(5) Save** button which saves changes made to this tab of the options. - - ![Access tab of GCP options menu showing the default settings.](./images/go-choose-folder/win/003-access-tab-default.png) - -1. Use the plus `+` and minus `-` buttons to add your research data folders and remove other folders, as needed. Click the "Shareable" checkbox next to each research data folder. Click "Save" when finished. - - In this example, we removed the default `C:/Users/%username%/Documents` folder with the minus `-` button and added the `D:/data` folder with the `+` button and check the "Shareable" box. You will want to pick the folder where your research data is stored. - - ![Access tab of GCP options menu showing new settings.](./images/go-choose-folder/win/004-access-tab-changed.png) - -1. Click the "General" tab. The "General" tab enables you to control some settings for the application itself and which folder is the default folder. The default folder will be the first one shown when accessing the Collection. - - - **(1) Run when Windows starts** checkbox enabling starting Globus Connect Personal when you start Windows. **Check this box if GCP should always be on when the computer is on.** - - **(2) Home Folder** text field that lets you choose which folder will be the default folder for your Collection. We recommend setting this to your primary shared folder from the previous step to simplify navigating your Collection in the Globus Web App. - - **(3) Save** button which saves changes made to this tab of the options. Be sure to click "Save" if you make changes here. - - ![General tab of GCP options menu showing default settings.](./images/go-choose-folder/win/005-general-tab-default.png) - -1. Check "Run when Windows starts" if needed. Change the "Home Folder" to match your research data folder. Click "Save" when done. - - In this example, we set the "Home Folder" to match the research data folder, `D:/data` we added in a previous step. If you have multiple research directories to share, you will need to choose just one for this field. Be sure to click save when you are done. - - ![General tab of GCP options menu](./images/go-choose-folder/win/006-general-tab-changed.png) - -To verify the existence and accessibility of your Collection proceed to [How Do I Find Collections I Created or Own?](#how-do-i-find-collections-i-created-or-own) - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -### Choose Specific Folders on MacOS - -1. In your MacOS notification area, locate the icon that looks like a small letter "g" in a circle. This is the icon for Globus Connect Personal. If you cannot locate the icon in the notification area, then open the Globus Connect Personal app on your computer and look for it again. - - ![Notification area showing icon of a small letter "g" in a circle.](./images/go-choose-folder/mac/001-notification-area.png) - -1. Right-click or command-click the icon to open the context menu. Click "Preferences…​". - - ![Context menu of Globus system tray icon showing preferences.](./images/go-choose-folder/mac/002-context-menu.png) - -1. A new window will appear with a tab labelled "Access". Click the "Access" tab if it is not already selected. In this "Access" tab is an interface to configure folders available on your GCP Collection. For most use cases, you should not check the writeable checkbox. Below is a summary of what each part of the menu does. - - - **(1) Accessible Directories and Files** table with "Directory or File", Shareable and Writeable columns. Any folder listed here will appear on your GCP Collection. Your research data folder or directories must be listed here to be shareable. - - - !!! note - - The terms Directories and Folders are synonyms here. - - - - - **(2) Shareable** column checkboxes controlling which folders can be shared with other users. Each of your research data directories must have this checkbox ticked to be shareable. **Check this box only if you want to share your data with others.** - - **(3) Writeable** column checkboxes controlling which folders can be written to by other users. If a folder is shared with other users, then they will be able to add, delete, or change the contents. We recommend against ticking these boxes for Research Cores serving data to customers. **Check this box only if you want others to be able to change your data.** - - **(4) Plus `+` and minus `-`** buttons that allow you to add or remove folders from the list. - - ![Access tab of GCP options menu showing the default settings.](./images/go-choose-folder/mac/003-access-tab.png) - -1. Use the plus `+` and minus `-` buttons to add your research data folders and remove other folders, as needed. Click the "Shareable" checkbox next to each research data folder. Click "Save" when finished. - -To verify the existence and accessibility of your Collection proceed to [How Do I Find Collections I Created or Own?](#how-do-i-find-collections-i-created-or-own) - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -## How Do I Find Collections I Created or Own? - -To find a Collection you own, use the following steps. - -1. Navigate to the [Globus Web App](#how-do-i-get-onto-the-globus-web-app) using your browser. You should be at the File Manager page. - - ![File Manager page of the Globus Web App](./images/common/file-manager.png) - -1. Click either of the Collection Search bar at the top of the File Manager page. This will take you to the Collection Search page. - - ![Collection Search page of the Globus Web App](./images/common/collection-search-page.png) - -1. Click the Your Collections tab to display a list of Collections you have created or own. - - ![Your Collections tab showing a GCP Collection.](./images/go-my-collections/003-your-collections.png) - -1. From here there are two options: - - 1. Click the name of the Collection to select it for a file transfer and be taken back to the File Manager page. - - ![File Manager page showing selected Collection on left-hand panel.](./images/go-my-collections/004-file-manager-selection-made.png) - - 1. Click the three dots icon at the right side of the entry to be taken to the Collection details page. - - ![Collectionm details page with overview tab selected.](./images/go-my-collections/005-collection-details-page.png) - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -## How Do I Enable Collection Sharing for My Globus Account? - -Before you can share Collections from your Globus Connect Personal (GCP) Collection with others, you must do a one-time setup for your account. You will need to join the "University of Alabama at Birmingham (HA)" (UAB HA) subscription group. Sharing any Collection requires a paid subscription with Globus. UAB Research Computing has a subscription, but Globus does not know your BlazerID is part of our subscription until you join the subscription group. So, Globus also does not know the GCP Collection you created is part of our subscription. By joining our UAB HA group, you and your GCP Collection are confirmed to be part of our subscription, and you can share Collections from the GCP Collection. - -To join the UAB HA group, we need to receive both a UAB support request, and a request to join the UAB HA group within the Globus Web App. We need both because we sometimes get spam applications through Globus. Having a Support Request helps us filter the spam. Please use the following steps to join. - -1. Submit a [Support Request](../../../help/support.md#how-do-i-create-a-support-ticket). In the request please include the following. - - - Your BlazerID. - - The text "Please add me to the Globus UAB HA subscription group." - - The reason you need to be able to share a Collection in Globus. For Research Cores, this would be to share data with your customers. - -1. In your browser [get onto the Globus Web App](#how-do-i-get-onto-the-globus-web-app). -1. In the left hand navigation panel click "Groups" to be taken to the Groups page. - - ![Globus groups page with no groups.](./images/go-uab-ha-group-access/001-groups-page.png) - -1. Ensure the "MY GROUPS" box is unchecked. In the "Filter groups" search bar enter "University of Alabama at Birmingham" to locate the "University of Alabama at Birmingham (HA)" (UAB HA) group. - - ![Globus group page showing search results with University of Alabama at Birmingham (HA) listed](./images/go-uab-ha-group-access/002-groups-search.png) - -1. Click on the UAB HA group name to be taken to the group details page. The Overview tab should be selected. - - ![UAB HA subscription group details page with overview tab selected.](./images/go-uab-ha-group-access/003-uab-ha-overview.png) - -1. Click on the "Join this Subscription" button to be taken to the form to submit a request to join. - - ![UAB HA subscription group join request form.](./images/go-uab-ha-group-access/004-join-form.png) - -1. Fill in the form fields and click the "Submit Application" button when completed. This will send you to a page notifying you that your membership is pending. A request has been sent to Research Computing, so please wait until you see a reply in the support request. - - ![Notice of pending membership.](./images/go-uab-ha-group-access/005-pending.png) - -1. When your membership has been accepted, you can verify by [returning to the Globus Web App](#how-do-i-get-onto-the-globus-web-app) and navigating to the Groups page. Check the "MY GROUPS". You should see "University of Alabama at Birmingham (HA)" listed with a green, circled checkmark and the word "ACTIVE". - - ![Groups page with my groups checked showing UAB HA subscription group](./images/go-uab-ha-group-access/006-my-groups.png.png) - -When you have completed the steps in this section, proceed to [creating a Collection](#how-do-i-create-a-collection). - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -## How Do I Create a Collection? - -There are three ways to create a Collection. - -- Start from [an existing Collection you created or own](#how-do-i-find-collections-i-created-or-own). -- [Install Globus Connect Personal](#how-do-i-install-globus-connect-personal) and [share a folder](#how-do-i-choose-specific-folders-to-share-using-globus-connect-personal) to create a Collection. -- Install and configure one or more Collections with Globus Connect Server. To do this, please [Contact Support](../../../help/support.md#how-do-i-create-a-support-ticket) to start a discussion. - -The instructions below assume you are starting from an existing Collection. The instructions will work to create a subset of your Globus Connect Personal Collection. - -1. [Get onto the Globus Web App](#how-do-i-get-onto-the-globus-web-app). -1. [Find the existing Collection](#how-do-i-find-collections-i-created-or-own) and access its details page. -1. Click the "Collection" tab. - - ![Overview tab of details page with Collection tab to the right.](./images/go-create-collection/001-collection-tab.png) - -1. Click the "+ Add Guest Collection" button to be taken to a form to create the new Collection. - - ![Collection tab of details page with Add Guest Collection button.](./images/go-create-collection/002-add-guest-collection.png) - -1. Fill out the form. - - - **(1) You Are Sharing**: Shows which Collection you will be sharing all or part of. Helpful to verify you are sharing the right Collection - - **(2) Path**: Which path you wish to share. By default, the "/" path is the root of the original Collection. This may not be appropriate for your use case. Generally, when creating one Collection from another, you will want to pick a subfolder of the original Collection. Do this with the "Browse" button. - - **(3) Display Name**: The name people will see when accessing this Collection, and what they will search for when looking for the Collection. - - **(4) Description**: An optional description for the Collection. - - **(5) Keywords**: Optional keywords to help locating the Collection. May be useful for broadly or publicly shared data. - - ![Collection creation form.](./images/go-create-collection/003-form.png) - -1. Click the "Create Guest Collection" button to create the Collection. You will be taken to the details page of the new Collection, on the Permissions tab. - - ![Permissions tab of new Collection details page.](./images/go-create-collection/004-completion.png) - -When you have created a Collection, you are ready to [share the Collection with others](#how-do-i-share-a-collection-with-others). - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -## How Do I Share a Collection With Others? - -Before sharing a Collection with others, you will need to first [create a Collection](#how-do-i-create-a-collection) you administer. If the Collection is a [Globus Connect Personal Collection](#how-do-i-install-globus-connect-personal), then you will also need to [enable Collection sharing for your Globus account](#how-do-i-enable-collection-sharing-for-my-globus-account). If these prerequisites have been met, then you are ready to setup a Collection to be shared with others. Please follow the instructions below. - -1. [Get onto the Globus Web App](#how-do-i-get-onto-the-globus-web-app). -1. [Find the Collection](#how-do-i-find-collections-i-created-or-own) you wish to share. In this case we are looking for the "customer-share" Collection. -1. On the "customer-share" Collection details page, click the Permissions tab. - - ![Collection details page overview tab.](./images/go-share-collection/001-collection-details-overview.png) - -1. Click the "Add Permissions – Share With" button. - - ![Collection details page permissions tab.](./images/go-share-collection/002-collection-details-permissions-before.png) - -1. Fill in the form. - - - **Path** defaults to the root of the Collection. This may or may not be acceptable for your use case. Be sure to limit access to only the data that needs to be shared. Use the file browser available in the form to find the correct level of access. For now grant permission to the default. - - **Share With**: radio buttons to provide control over who to share with. In almost all cases, you will want to share with a single user, which is what we will show here. - - **User**: Use the "User" search bar to find the specific user you wish to share with. If they have never accessed Globus before, they will not appear in the search results. You can safely enter their email address to add them anyway. For now grant permission to yourself for the purposes of experimenting. - - **Email Notification** checkbox. We recommend sending an email notification as a convenience. - - **To** entry field: who to send the email to. We recommend the same person as the "User" selected earlier. There may be future cases where you want to notify others, such as a supervisor, as well. - - **Message** text entry field: the optional content to send in the email message. - - **Permissions** The "read" permission must be granted, as that is the point of sharing the Collection. You may additionally give "write" permission to create a two-way collaboration. We recommend Research Cores not grant "write" permission. If you are using a Globus Connect Personal Collection, then "write" permission requires you to correctly [configure your Collection](#how-do-i-choose-specific-folders-to-share-using-globus-connect-personal) to make your shared folder writable. - - ![Add permissions form.](./images/go-share-collection/003-add-permissions-form.png) - -1. Click the "Add Permission" button to grant permission. You should see a notification confirming the permissions granted. At this point permissions have been granted and the Collection is shared with another person. If you need to add more people, click the "Add another Permision" button and repeat the process. Otherwise click "Done". For now click "Done". - - ![Permission confirmation notification.](./images/go-share-collection/004-permissions-confirmation.png) - -1. When you click "Done" you should be taken back to the Permissions tab of the Customer Share page. You should see a new entry with "Path: /". If you click the drop-down arrow you will see yourself listed with "Read" permission. If you need to revoke permissions, return to this page and click the icon that looks like a trash can. - - ![Collection details page permissions tab showing new entry.](./images/go-share-collection/005-colection-details-permissions-after.png) - -[Return to the top](#globus-tutorials-for-uab-research-organizations). - -## How Do I Share Data With a Research Core Customer? - -If you manage data for a Research Core, please [Contact Support](../../../help/support.md#how-to-request-support) to start a conversation. The answer to this question is currently being developed, so we will need to work together to find the ideal solution. - -[Return to the top](#globus-tutorials-for-uab-research-organizations). diff --git a/docs/data_management/transfer/tutorial/images/gi-find-shared-collections/004-selected.png b/docs/data_management/transfer/tutorial/images/gi-find-shared-collections/004-selected.png deleted file mode 100644 index b3aa4e829..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-find-shared-collections/004-selected.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-cheaha/005-ready-to-select.png b/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-cheaha/005-ready-to-select.png deleted file mode 100644 index 70bbada75..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-cheaha/005-ready-to-select.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-cheaha/008-selection.png b/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-cheaha/008-selection.png deleted file mode 100644 index 42a1d9d8f..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-cheaha/008-selection.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-cheaha/009-transfer-started.png b/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-cheaha/009-transfer-started.png deleted file mode 100644 index 31cca7870..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-transfer-collection-to-cheaha/009-transfer-started.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/images/gi-transfer-lts-to-cheaha/005-ready-to-select.png b/docs/data_management/transfer/tutorial/images/gi-transfer-lts-to-cheaha/005-ready-to-select.png deleted file mode 100644 index c4cf9e7ea..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-transfer-lts-to-cheaha/005-ready-to-select.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/images/gi-transfer-lts-to-cheaha/008-selection.png b/docs/data_management/transfer/tutorial/images/gi-transfer-lts-to-cheaha/008-selection.png deleted file mode 100644 index fac8ccbff..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-transfer-lts-to-cheaha/008-selection.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/images/gi-transfer-lts-to-cheaha/009-transfer-started.png b/docs/data_management/transfer/tutorial/images/gi-transfer-lts-to-cheaha/009-transfer-started.png deleted file mode 100644 index 06d1d9174..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-transfer-lts-to-cheaha/009-transfer-started.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/images/gi-uab-collections/001-cheaha.png b/docs/data_management/transfer/tutorial/images/gi-uab-collections/001-cheaha.png deleted file mode 100644 index ae912699a..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-uab-collections/001-cheaha.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/images/gi-web-app/001-login.png b/docs/data_management/transfer/tutorial/images/gi-web-app/001-login.png deleted file mode 100644 index d8080c607..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-web-app/001-login.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/images/gi-web-app/002-search.png b/docs/data_management/transfer/tutorial/images/gi-web-app/002-search.png deleted file mode 100644 index a0d9f9241..000000000 Binary files a/docs/data_management/transfer/tutorial/images/gi-web-app/002-search.png and /dev/null differ diff --git a/docs/data_management/transfer/tutorial/index.md b/docs/data_management/transfer/tutorial/index.md deleted file mode 100644 index 45f48c0a6..000000000 --- a/docs/data_management/transfer/tutorial/index.md +++ /dev/null @@ -1,13 +0,0 @@ -# Data Transfer Tutorials - -## Globus - -To learn how to use Globus, everyone should start with the [Globus for Individuals Tutorials](./globus_individual_tutorial.md) to become familiar with how to get on Globus and how it works. - -If you manage a lab or a Research Core on campus, you may also be interested in our [Globus for Organizations Tutorials](./globus_organization_tutorial.md) and [Globus Group Management](./globus_group_management.md). Research Core managers and directors should please [Contact Support](../../../help/support.md) to start a conversation around Data Management good practices. - -Tutorials: - -- [How to use Globus for Individuals](./globus_individual_tutorial.md) -- [How to use Globus for Organizations such as Labs and Cores](./globus_organization_tutorial.md) -- [How to manage Globus Groups](./globus_group_management.md) diff --git a/docs/education/courses.md b/docs/education/courses.md index baf52cb6b..57737bcf7 100644 --- a/docs/education/courses.md +++ b/docs/education/courses.md @@ -24,4 +24,4 @@ If you have questions about the course please feel free to reach out to [Support ### Syllabus -The syllabus is available as a public webpage at , hosted on [LTS](../data_management/lts/index.md). +The syllabus is available as a public webpage at , hosted on [LTS](../data_management/storage/lts/index.md). diff --git a/docs/help/support.md b/docs/help/support.md index 6c5bc0743..e38f8694b 100644 --- a/docs/help/support.md +++ b/docs/help/support.md @@ -82,7 +82,7 @@ Select the option appropriate for your need. The "View all open tickets" will sh ## How Do I Request or Change Shared Storage? -Please see our [Storage page](../data_management/index.md) for more information. +Please see our [Storage page](../data_management/storage/index.md) for more information. ## How Do I Request New Software Installed? diff --git a/docs/index.md b/docs/index.md index 49a7f478c..a79501146 100644 --- a/docs/index.md +++ b/docs/index.md @@ -26,8 +26,8 @@ Create [your Cheaha Account](./account_management/cheaha_account.md), then check - [PyTorch and Tensorflow on Cheaha with Anaconda](./cheaha/tutorial/pytorch_tensorflow.md) - [Getting Started with Cloud.RC for On-Premises Cloud Computing](./uab_cloud/tutorial/index.md) - Data Management - - [How Do I Use Globus for Data Transfer?](./data_management/transfer/tutorial/index.md) - - [What Type of Storage Do I Need?](./data_management/index.md#what-type-of-storage-do-i-need) + - [How Do I Use Globus for Data Transfer?](./data_management/transfer/globus/index.md) + - [What Type of Storage Do I Need?](./data_management/storage/index.md#what-type-of-storage-do-i-need) ### Success Stories diff --git a/docs/news/posts/2025-04-07-reduced-lts-transfer-speeds-on-globus.md b/docs/news/posts/2025-04-07-reduced-lts-transfer-speeds-on-globus.md index a394ecfa7..e42b5a040 100644 --- a/docs/news/posts/2025-04-07-reduced-lts-transfer-speeds-on-globus.md +++ b/docs/news/posts/2025-04-07-reduced-lts-transfer-speeds-on-globus.md @@ -10,6 +10,6 @@ We are aware of increased transfer times into and out of LTS when using Globus. -At this time, we recommend preferring to use [`s5cmd`](../../data_management/lts/interfaces.md#s5cmd) to transfer large amounts of data into and out of LTS. +At this time, we recommend preferring to use [`s5cmd`](../../data_management/storage/lts/interfaces.md#s5cmd) to transfer large amounts of data into and out of LTS. Please feel free to [Contact Us](../../help/support.md) for more information or alternative solutions. diff --git a/docs/news/posts/2025-10-07-migration-overview.md b/docs/news/posts/2025-10-07-migration-overview.md index 01902eb4c..e074bc659 100644 --- a/docs/news/posts/2025-10-07-migration-overview.md +++ b/docs/news/posts/2025-10-07-migration-overview.md @@ -94,7 +94,7 @@ To effectively manage growing storage needs while controlling the costs associat ### What GPFS Tiering IS NOT 1. **Tiering is NOT a replacement for LTS**. - 1. Data you know you will not use for an extended period of time should be moved to [LTS](../../data_management/lts/index.md). + 1. Data you know you will not use for an extended period of time should be moved to [LTS](../../data_management/storage/lts/index.md). 1. Storage quotas will be enforced regardless of a given file's storage tier. This means for a standard project quota, only 25 TiB can be stored in a project across both GPFS and Ceph. 1. **Tiering is NOT a backup** 1. A file's data will only exist on one tier, CephFS or GPFS, **never on both**. @@ -109,7 +109,7 @@ To effectively manage growing storage needs while controlling the costs associat !!! critical - It is imperative to understand that tiered storage does not equal a backup. We do not provide a traditional, automatic backup for data stored on Cheaha. All data are erasure-encoded in case of hardware failure, they are not backed up in case of user error. Please see information about [LTS](../../data_management/lts/index.md) for a potential backup solution. + It is imperative to understand that tiered storage does not equal a backup. We do not provide a traditional, automatic backup for data stored on Cheaha. All data are erasure-encoded in case of hardware failure, they are not backed up in case of user error. Please see information about [LTS](../../data_management/storage/lts/index.md) for a potential backup solution. ### What CephFS IS diff --git a/docs/uab_cloud/sharing_cloud_environment.md b/docs/uab_cloud/sharing_cloud_environment.md index f2d86a2d5..aeeee69ef 100644 --- a/docs/uab_cloud/sharing_cloud_environment.md +++ b/docs/uab_cloud/sharing_cloud_environment.md @@ -16,7 +16,7 @@ The benefits of creating and using a Shared Cloud Environment for your Lab/Core ## How Do I Create a Shared Cloud Environment for My Lab/Core -To use the shared cloud resources available, you will need to send in a request to the UAB IT Research Computing Team via email . In the email, please state clearly your needs and the resources you would require for your lab. Your request should also include members of your lab to be included in the Shared Cloud Environment, and a preferred name (usually same as lab project folder on Cheaha). Please follow the naming conventions for requesting a [shared storage](../data_management/index.md#how-do-i-request-shared-storage) on our platforms in your request. +To use the shared cloud resources available, you will need to send in a request to the UAB IT Research Computing Team via email . In the email, please state clearly your needs and the resources you would require for your lab. Your request should also include members of your lab to be included in the Shared Cloud Environment, and a preferred name (usually same as lab project folder on Cheaha). Please follow the naming conventions for requesting a [shared storage](../data_management/storage/index.md#how-do-i-request-shared-storage) on our platforms in your request. ## How Do I Switch Project Spaces? diff --git a/mkdocs.yml b/mkdocs.yml index 0f83ff2eb..cc92783c7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -82,14 +82,16 @@ plugins: # order matters! cheaha/open_ondemand/ood_jupyter.md: cheaha/open_ondemand/ood_jupyter_notebook.md cheaha/slurm/sbatch_usage.md: cheaha/slurm/submitting_jobs.md contributor_guide.md: contributing/contributor_guide.md - data_management/cheaha_gpfs_storage.md: data_management/cheaha_storage_gpfs/index.md - data_management/LTS/lts.md: data_management/lts/index.md - data_management/LTS/sharing.md: data_management/lts/iam_and_policies.md - data_management/lts/lts.md: data_management/lts/index.md - data_management/lts/policies.md: data_management/lts/iam_and_policies.md - data_management/lts/sharing.md: data_management/lts/iam_and_policies.md - data_management/cheaha_storage_gpfs/project_directory_organization.md: data_management/cheaha_storage_gpfs/project_directories.md#project-directory-organization - data_management/storage.md: data_management/index.md + data_management/cheaha_gpfs_storage.md: data_management/storage/cheaha_storage_gpfs/index.md + data_management/cheaha_storage_gpfs/project_directory_organization.md: data_management/storage/cheaha_storage_gpfs/project_directories.md#project-directory-organization + data_management/LTS/lts.md: data_management/storage/lts/index.md + data_management/LTS/sharing.md: data_management/storage/lts/iam_and_policies.md + data_management/lts/lts.md: data_management/storage/lts/index.md + data_management/lts/policies.md: data_management/storage/lts/iam_and_policies.md + data_management/lts/sharing.md: data_management/storage/lts/iam_and_policies.md + data_managment/transfer/tutorial/globus_individual_tutorial.md: data_management/transfer/globus/globus_individual_tutorial.md + data_managment/transfer/tutorial/globus_organization_tutorial.md: data_management/transfer/globus/globus_organization_tutorial.md + data_managment/transfer/tutorial/globus_group_management.md: data_management/transfer/globus/globus_group_management.md environment_management/anaconda_environments.md: workflow_solutions/using_anaconda.md environment_management/containers.md: workflow_solutions/getting_containers.md environment_management/git.md: workflow_solutions/git_collaboration.md @@ -125,35 +127,39 @@ nav: - Leaving UAB: account_management/leaving_uab.md - Data Management: - data_management/index.md - - Cheaha Storage (GPFS): - - data_management/cheaha_storage_gpfs/index.md - - User and Home Directories: data_management/cheaha_storage_gpfs/individual_directories.md - - Project Directories: data_management/cheaha_storage_gpfs/project_directories.md - - Network Scratch: data_management/cheaha_storage_gpfs/network_scratch.md - - Local Scratch: data_management/cheaha_storage_gpfs/local_scratch.md - - Temporary Files (/tmp/ Directory): data_management/cheaha_storage_gpfs/temporary_files.md - - Storage Alternatives: data_management/alternate_storage.md - - Long Term Storage: - - data_management/lts/index.md - - Tutorial: - - data_management/lts/tutorial/index.md - - Individual LTS and s3cmd Workflow: data_management/lts/tutorial/individual_lts_tutorial.md - - Shared LTS and s3cmd Workflow: data_management/lts/tutorial/shared_lts_tutorial.md - - Interfacing With LTS: data_management/lts/interfaces.md - - Identity and Access Management: data_management/lts/iam_and_policies.md - - UAB Core Accounts: data_management/lts/lts_cores.md - - FAQ: data_management/lts/lts_faq.md + - Storage: + - data_management/storage/index.md + - Cheaha Storage (GPFS): + - data_management/storage/cheaha_storage_gpfs/index.md + - User and Home Directories: data_management/storage/cheaha_storage_gpfs/individual_directories.md + - Project Directories: data_management/storage/cheaha_storage_gpfs/project_directories.md + - Network Scratch: data_management/storage/cheaha_storage_gpfs/network_scratch.md + - Local Scratch: data_management/storage/cheaha_storage_gpfs/local_scratch.md + - Temporary Files (/tmp/ Directory): data_management/storage/cheaha_storage_gpfs/temporary_files.md + - Long Term Storage: + - data_management/storage/lts/index.md + - Tutorial: + - data_management/storage/lts/tutorial/index.md + - Individual LTS and s3cmd Workflow: data_management/storage/lts/tutorial/individual_lts_tutorial.md + - Shared LTS and s3cmd Workflow: data_management/storage/lts/tutorial/shared_lts_tutorial.md + - Interfacing With LTS: data_management/storage/lts/interfaces.md + - Identity and Access Management: data_management/storage/lts/iam_and_policies.md + - UAB Core Accounts: data_management/storage/lts/lts_cores.md + - FAQ: data_management/storage/lts/lts_faq.md + - Code Storage: data_management/storage/code_storage.md + - Storage Alternatives: data_management/storage/alternate_storage.md - Data Transfer Tools: - data_management/transfer/index.md - - Tutorial: - - data_management/transfer/tutorial/index.md - - Globus for Individuals: data_management/transfer/tutorial/globus_individual_tutorial.md - - Globus for Organizations: data_management/transfer/tutorial/globus_organization_tutorial.md - - Globus Group Management: data_management/transfer/tutorial/globus_group_management.md - - Globus: data_management/transfer/globus.md + - Globus: + - data_management/transfer/globus/index.md + - Getting Started: data_management/transfer/globus/login_to_globus.md + - Install Globus Connect Personal: data_management/transfer/globus/gcp_install.md + - Set Up Globus Connect Personal: data_management/transfer/globus/gcp_setup.md + - Globus for Individuals: data_management/transfer/globus/globus_individual_tutorial.md + - Globus for Organizations: data_management/transfer/globus/globus_organization_tutorial.md + - Globus Group Management: data_management/transfer/globus/globus_group_management.md - RClone: data_management/transfer/rclone.md - FileZilla: data_management/transfer/filezilla.md - - Code Storage: data_management/code_storage.md - Research Data Responsibilities: data_management/research_data_responsibilities.md - Workflow Solutions: - Using the Shell: workflow_solutions/shell.md