How to Improve Job Performance Using Scratch on HPCC

To optimize your job performance on the HPCC, it recommend using faster SSD storage directories, such as /scratch, /scratch2, or /scratch3, for your temporary working files. These directories are equipped with Solid State Drives (SSD), which provide faster access times and superior data transfer rates.

Each of these directories offers 5TB of storage space, accommodating the significant amount of temporary output that your programs may generate during execution.

Scratch Storage Information

Feature	Details
Directory	/scratch, /scratch2, or /scratch3
User Capacity	5TB per storage directory
Clean-up Policy	Files not accessed in the past 7 days are subject to system clean-up
Usage	Short term High performance Good for large amount of temporary files assessed during job execution Good for large intermediate output files from jobs, especially parallel jobs

To utilize scratch storage effectively, follow these example steps:

1. Creates a directory for the job in the scratch directory

mkdir -p /scratch/$USER/job1

2. Copies all files and directories from the source (/data/home/$USER/job1) to the scratch directory

cp -a /data/home/$USER/job1/* /scratch/$USER/job1/

3. Changes the working directory to the scratch job directory and run your job

cd /scratch/$USER/job1
./your_script_or_command_here

4. Copies the results back to the home directory after the job completes

cp -a /scratch/$USER/job1/results/* /data/home/$USER/job1/results/

5. Deletes the scratch directory to free up space after the job is done.

rm -rf /scratch/$USER/job1

 
Before running the example commands, please ensure that you update the folder directory paths to match your specific environment. The paths /data/home/$USER and /scratch/$USER are placeholders and should be replaced with the actual paths where your data is stored and where you want to perform your computations. For instance, if your data is located in /home/$USER/project1 and you want to use /scratch/$USER/project1 as your scratch directory, update the paths accordingly. Failing to modify these paths may result in errors or unintended file operations. Always double-check the paths to ensure they point to the correct locations.