Automatically calibrate spark.sql.shuffle.partitions or spark.default.parallelism
Automatically calibrate spark.sql.shuffle.partitions and spark.default.parallelism.
Spark will give the user the option to find the optimal range of values for: spark.sql.shuffle.partitions and spark.default.parallelism parameters. Also a good solution will be to print a suggested value to the log. For example in case of OOM cause by a low number of big partitions spark driver can log a suggestion to a better value for those parameters.
5
votes