Configuring a Job for Dynamic Resource Allocation
There are two ways to customize dynamic resource allocation properties for a specific job:
Include property values in the
spark-submitcommand, using the-confoption.This approach loads the default
spark-defaults.conffile first, and then applies property values specified in yourspark-submitcommand. Here is an example:spark-submit --conf “property_name=property_value”Create a job-specific
spark-defaults.conffile and pass it to thespark-submitcommand.This approach uses the specified properties file, without reading the default property file. Here is an example:
spark-submit --properties-file <property_file>

