Allow auto liquid clustering

[ShaneMazur]

Resolves #932

Description

Adds config auto_liquid_cluster to allow for tables to be CLUSTER BY AUTO on creation or update (i.e. incremental run)

Checklist

• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[benc-db]

Can you do me one small favor: do you know if Databricks has any documentation that covers this? I'm struggling to find it...I'm fairly confident that if the E2E tests pass, this is valid Databricks SQL, but would be nice to have.

[benc-db]

Can you do me one small favor: do you know if Databricks has any documentation that covers this? I'm struggling to find it...I'm fairly confident that if the E2E tests pass, this is valid Databricks SQL, but would be nice to have.

Part of why I'm interested in the documentation is to see if we are using the most discoverable name for the config.

[benc-db]

Are you set up with hatch in your dev environment? If so, please run hatch run code-quality to fix the linting issue. If you don't, I can do it.

[ShaneMazur]

Can you do me one small favor: do you know if Databricks has any documentation that covers this? I'm struggling to find it...I'm fairly confident that if the E2E tests pass, this is valid Databricks SQL, but would be nice to have.

Part of why I'm interested in the documentation is to see if we are using the most discoverable name for the config.

There are these but nothing in the Databricks official documentation but it is mentioned in this video from last year's summit

There was also a PDF provided to us by the PM but not sure if that has changed and probably not a good idea to share it here anyway. Let me know and I can share it if necessary.

[ShaneMazur]

Are you set up with hatch in your dev environment? If so, please run hatch run code-quality to fix the linting issue. If you don't, I can do it.

Done ✅

[benc-db]

No worries, it just means that this will likely be a 'secret' feature of the adapter until the Databricks feature is documented. When you set auto cluster by, do you still need to run optimize manually? Going to pull and push to run the E2E tests.

[ShaneMazur]

No worries, it just means that this will likely be a 'secret' feature of the adapter until the Databricks feature is documented. When you set auto cluster by, do you still need to run optimize manually? Going to pull and push to run the E2E tests.

Yes - CLUSTER BY AUTO is just for automatic key selection. Triggering the clustering automatically is still handled by predictive optimization.

Enabling it for your Databricks workspace is hidden in the account UI under Settings > Feature enablement:

[benc-db]

Do you have an environment where you could write/run an E2E test? Unit test looks good, but would like to see a test similar to https://github.com/databricks/dbt-databricks/blob/main/tests/functional/adapter/liquid_clustering/test_liquid_clustering.py
to validate that the inserted code doesn't break in DBSQL.

[benc-db]

Please add a 'functional' aka E2E test similar to existing liquid clustering tests. Since liquid clustering is an optimization thing, we can't really validate the outcomes, but I just want a regression test to ensure we notice if they change syntax for auto cluster by before final release.

[ShaneMazur]

Please add a 'functional' aka E2E test similar to existing liquid clustering tests. Since liquid clustering is an optimization thing, we can't really validate the outcomes, but I just want a regression test to ensure we notice if they change syntax for auto cluster by before final release.

Added that and included auto liquid clustering in the optimize logic since it is still valid to run optimize on liquid clustering with auto selected keys

[benc-db]

Added that and included auto liquid clustering in the optimize logic since it is still valid to run optimize on liquid clustering with auto selected keys

Looks good, let me just run tests. If everything is green, I'll merge it in. Thanks!