Following up on Databricks Performance Tuning with the best place to start: allocating Spark clusters. If you don't allocate sufficient resources, nothing else will fix the problem. How many nodes? How large should the driver and workers be? Do you need GPUs or CPUs? Should you use Photon? These and many more questions will be covered in detail.
Support me on Patreon
https://www.patreon.com/bePatron?u=63260756
Slides
https://github.com/bcafferky/shared/blob/master/DatabricksPerfTuning/lesson02_DatabricksPerfTuningHardware.pdf
Referenced Video by Daniel Tomes, Databricks
https://www.youtube.com/watch?v=daXEp4HmS-E&t=2111s
See the Full Playlist Here:
https://www.youtube.com/playlist?list=PL7_h0bRfL52qw5SkSI_P_kURU2njM1UO-