Limitations When Using DistCp with S3
When using DistCp with data in S3, consider the following limitations:
The
-appendoption is not supported.The
-diffoption is not supported.The
-atomicoption causes a rename of the temporary data, so significantly increases the time to commit work at the end of the operation. Furthermore, as S3A does not offer atomic renames of directories, the-atomic operation doesn't actually deliver what is promised.Avoid using this option.All
-poptions, including those to preserve permissions, user and group information, attributes checksums, and replication are ignored.CRC checking will not be performed, irrespective of the value of the
-skipCrcflag.

