ICTO - FAQ » Research Computing » HPC Cluster » Best Practices for Copying Data on HPCC

Best Practices for Copying Data on HPCC

Print
When working with shared data folders, especially when multiple users are copying data simultaneously, choosing the right command for file operations is crucial. To improve the stability and efficiency of the data folder, it is recommended to use the “cp” command instead of “rsync” or “mv”.

Why Use the “cp” Command?

The “cp” command is straightforward and performs a basic file copy operation without any additional synchronization or verification steps. This simplicity can enhance stability, especially when dealing with multiple users, large amounts of data, or complex directory structures. Additionally, it avoids the overhead of comparing and synchronizing files, making it faster for simple copying operations.

Using “cp” ensures that the original data remains intact in the source directory, whereas “mv” moves (and deletes) the data, which can be risky.

How to Use the “cp” Command

To see more options and details:

cp --help

Example: Copying Data to Another Folder

cp -av /home/$USER/source_directory/* /home/$USER/destination_directory/

Example: Copying Data to or from the Shared Project Folder 
* Specially for FHS, to ensure new files inherit the group of the shared folder

cp -Rv /home/$USER/source_directory/* /home/MyLab/sharedata/

Options Explained:

  • -a: Copy files and directories recursively and preserve attributes (timestamps, permissions, ownership)
  • -R: Copy directories recursively.
  • -v: Show detailed output of the copy process.
Tagged: