Disk Space Troubleshooting: Deleted Files

Overview

This lab teaches a critical troubleshooting technique for Linux system administrators: handling disk space issues when deleted files continue to consume space because they’re still open by running processes.

Lab Objectives

After completing this lab, you will be able to:

• Diagnose disk space issues using df and du

• Understand how Linux handles file deletion and file descriptors

• Use lsof to find deleted files that are still open

• Identify processes holding deleted files open

• Resolve disk space issues by properly closing filehandles

Scenario

You’re working at Super-Business where Manager Scott has reported a critical disk space problem on a production server. You’ll need to investigate the issue, find what’s consuming the space, clean it up, and handle an unexpected complication that arises. This teaches the complete troubleshooting workflow from problem identification through resolution.

Getting Started

To run this lab locally for development:

• Use the utilities in the utilities/ directory

• Run ./utilities/lab-build to build the documentation

• Run ./utilities/lab-serve to preview locally

Understanding the Basic Template Directory Structure

1. Content Directory

   tree output

   .
   ├── README.adoc                 # Readme file
   ├── config                      # Defines VMs, network and firewall
   ├── runtime-automation          # Scripts directory for setup/next/solve modules
   ├── setup-automation            # Scripts directory when provisioning
   ├── ui-config.yml               # Configuration file
   ├── site.yml                    # Theme config
   └── content                     # Content directory
       ├── antora.yml              # Antora configuration file. You can add "inline vars" here to render within your content
       └── modules/ROOT
           ├── assets
           │   └── images                       # Images used in your content
           │       └── example-image.png
           ├── pages                            # Your content goes here
           │   ├── module-01.adoc               # First module of your lab, e.g. overview etc
           │   ├── module-02.adoc               # Second module of your lab
           │   └── module-03.adoc               # Third module

2. Lab definition directory

   tree output

   config/
   ├── firewall.yaml                           # Define the egress/ingress rules to be allowed
   ├── instances.yaml                          # Define the instances to be created
   └── networks.yml                            # Define the networks to be created

3. Runtime Scripts Directory Structure

   tree output

   runtime-automation/
   ├── module_1                            # The directory name should be the same as
   │                                       # the module content page name
   │   ├── setup.sh                        # Setup playbook which run at start of module
   │                                       # and when clicked on next button
   │   ├── solve.sh                        # Solve playbook, run when clicked on solve button
   │   └── validation.sh                   # Validation playbook, run when click on next button
   ├── ansible.cfg                         # Ansible configuration used with module playbooks
   └── main.yml                            # Playbook to be called in each step

4. Setup Scripts Directory Structure

   tree output

   setup-automation/
   ├── setup-vmname.sh                     # Script to be executed on vmname when is provisioned
   ├── ansible.cfg                         # Ansible configuration used with module playbooks
   └── main.yml                            # Playbook to be called in each step

Understanding UI Configration file

1. Example: The ui-config.yml configuration file

   tree output

   antora:
     name: modules
     dir: www
     modules:
       - name: 01-playbook-inventory               # Module Name
         label: Playbook Inventory                 # Module Label
         solveButton: true                         # Enable Solve button
       - name: 02-playbook-directory-structure
         label: Playbook Directory Structure
         solveButton: true
   tabs:
     - name: VSCode Editor                         # Tab name
       port: 8443                                  # Port no of the service
       path: /?folder=/home/rhel/ansible-files/    # Path suffix
     - name: ">_control"                           # Tab name
       port: 443                                   # Port of the tt1 terminal
       path: /tty1                                 # Path to the tty1 terminal

Development Cycle

1. Edit your content in content/modules/ROOT/pages/

2. Use git to branch and commit your work

3. Push your work to your repo

   1. You should use git tags or git branches in production

   2. However development items default to the head of main