Configuring a Job for Dynamic Resource Allocation
There are two ways to customize dynamic resource allocation properties for a specific job:
Include property values in the
spark-submitcommand, using the-confoption.This approach will load the default
spark-defaults.conffile first, and then apply property values specified in yourspark-submitcommand. Here is an example:spark-submit —conf “property_name=property_value”Create a job-specific
spark-defaults.conffile and pass it to thespark-submitcommand.This approach will use the specified properties-file, without reading the default property file.
spark-submit —properties-file <property_file>

