03 - Auto Optimize

Auto Optimize is a feature in Databricks that helps manage and optimize Delta Lake tables by automatically handling file sizes and data organization. This feature is designed to improve query performance and reduce the need for manual maintenance tasks.

What is Auto Optimize?

Auto Optimize is a feature in Databricks that automates the optimization of Delta Lake tables by managing the size and organization of data files. It consists of two main components:

Auto Compaction: Automatically compacts small files into larger ones during write operations. Optimize Write: Automatically optimizes the size of files written to Delta Lake tables to ensure they are of an optimal size for query performance. (Attempts to write 128 MB Files)

enable_auto_optimize