S3 Performance Checklist
Use this checklist to ensure optimal performance when working with data in S3.
Checklist for Data
[ ]Amazon S3 bucket is in same region as the EC2-hosted cluster. Learn more[ ]The directory layout is "shallow". For directory listing performance, the directory layout prefers "shallow" directory trees with many files over deep directory trees with only a few files per directory.[ ]The "pseudo" block size set infs.s3a.block.sizeis appropriate for the work to be performed on the data.[ ]Copy to HDFS any data that needs to be repeatedly read to HDFS.
Checklist for Cluster Configs
[ ]Setyarn.scheduler.capacity.node-locality-delayto 0 to improve container launch times.[ ]When copying data using DistCp, use the following performance optimizations.[ ]If planning to use Hive with S3, review Improving Hive Performance with Cloud Object Stores.[ ]If planning to use Spark with S3, review Improving Spark Performance with Cloud Object Stores.
Checklist for Code
[ ]Application does not makerename()calls. Where it does, it does not assume the operation is immediate.[ ]Application does not assume thatdelete()is near-instantaneous.[ ]Application usesFileSystem.listFiles(path, recursive=true)to list a directory tree.[ ]Application prefers forward seeks through files, rather than full random IO.[ ]If making "random" IO throughseek()andread()sequences or and Hadoop'sPositionedReadableAPI,fs.s3a.experimental.input.fadviseis set torandom. Learn more
More

