How to Copy Large Number of Files in Linux

We use the cp command in Linux to copy files and directories from one directory to another. It can be simply used to copy a few files or directories, or it can be used with the '-r' argument (which stands for ‘recursive‘) to copy a directory and the whole directory tree structure underneath it.

The syntax to use to cp command is:

$ cp <file1> <file2> ... <fileN> <target_directory>
$ cp -r <directory1> <directory2> ... <directoryN> <target_directory>
$ cp file1 file2 file3 /tmp

If a large number of files are to be copied, they can be specified with wildcards; if they are named in a sequential pattern (Eg. filename_1, filename_2, etc.) or if there are a large number of files of the same extension (Eg. .mp4).

$ cp *.txt /tmp
$ cp file_* /tmp

The cp command can thus have any number of arguments, which are either files, directories or both. The last argument is the target directory where they are to be copied to.

Copying Large Number of Files: “Argument list too long” Error

While theoretically ‘cp‘ can have as many arguments as possible, there is a predefined limit on the maximum number of arguments that a single command can work with. This is a limit defined in the Linux system and is based on the stack size of the system.

Thus, if a very high number of files are being specified with the wildcard with ‘cp‘, Eg. over a hundred thousand files, it throws an error: “Argument list too long“.

For example, as shown in the image below, the folder contains 100002 files. Thus, it gives the aforementioned error when we run the copy command passing all the files with a wildcard.

$ ls -l | wc -l
$ cp file_* target/

Solution: Use ‘Find’ Command to Fix “Argument list too long” Error

To solve this problem, we make use of the find command, which basically searches for files in directories based on different parameters. We run ‘find‘ in the same directory as the files we want to copy.

Then we make use of the '-exec' parameter of ‘find‘ which will allow you to run any command over the output of find.

$ find . -maxdepth 1 -name "<filename_pattern>" -exec cp {} <target_directory> \;

Here, the first argument, '.' (current directory) is the directory where to find the files. The argument '-maxdepth 1' considers only the files and directories in the current directory. Without specifying this depth, ‘find‘ command will look through all subdirectories unnecessarily.

The argument "-name" and the string after that specifies the expression using which required files can be listed. Then we specify the ‘cp‘ command after argument '-exec'. The open brackets, '{ }' are a placeholder for the argument which is to be used from output of find.

$ find . -maxdepth 1 -name "file_*" -exec cp {} target/ \;

We can see that the files have been copied successfully to '/target'.

$ cd target/
$ ls | head -10
$ ls -l | wc -l

Similarly, if there are directories that are to be copied along with files, we run ‘cp‘ with the '-r' flag in the example below.

$ find . -maxdepth 1 -name "file_*" -exec cp -r {} target/ \;

Note that, in such a case, the directory name should also be searchable with the ‘filename pattern‘.

Related Read: How to Move Large Number of Files in Linux

Conclusion

In this article, we have seen how to copy a large number of files from one directory to another in Linux and thus how to workaround the issue ‘Argument list too long‘.

If you have any feedback or questions, let us know in the comments below!