Using DistCp with Azure ADLS and WASB
The
-appendoption is not supported.The
-diffoption is not supported.The
-atomicoption causes a rename of the temporary data, which slows down the upload. Avoid using this option.ADLS implements the same permissions model as HDFS, so some of the
-poptions work.WASB supports getting and setting the permissions, but these permissions do not control access to the data.. What they can do is ensure is that permissions can be restored after a back-up.
You can tune
fs.azure.selfthrottling.read.factorandfs.azure.selfthrottling.write.factor. Refer to Maximizing HDInsight throughput to Azure Blob Storage blog post.

